Skip to main content


There simply is no established or easy way to detect backdoors done the #xz way. We give powers and trust to maintainers because that is the development model.

Anyone suggesting there is an easy fix has not understood the issues at hand.

But we are Open Source which allows everyone to dig, check, read code and investigate.

#xz
in reply to daniel:// stenberg://

there is an infinite amt of work fixing this from the 'code' end ... we might need new, better theories of network computing ( which does not enable autocratic dictators ) ... operationally we should make it very explicit when a binary does something it is not expected to do. Changing from treating each vulnerability as exotica to 'cattle' is a difficult challenge ... because scale, scale, scale is going to overwhelm all good faith efforts.
in reply to Jim Fuller

@jimfuller I grant you that we can theorize about future possible fixes if given limitless amounts of resources, but I don't think that's very helpful
in reply to daniel:// stenberg://

thats my point -we do not have limitless resources ... to fix 'all the code' one by one, especially in that we have no model for funding such resources ... I do not claim to have a solution ...
in reply to daniel:// stenberg://

Yeah I think if there's anything this results in it will be that non-developer corporate people (hey Legal department) will realize that the fact that open source contributions are peer reviewed doesn't mean bad faith code cannot be merged.

... and that it doesn't matter if you're a paying Red Hat customer. You'll still end up with it in your product.

I do wonder if this will create some anti open source backlash though. It's of course not rational, but when has such decisions ever been.

in reply to Troed Sångberg

@troed I can imagine how this kind of attack can be done in closed sourced applications as well, with outsourcing, consultants etc. A good developer turned malicious is extremely hard to protect against.
in reply to daniel:// stenberg://

Absolutely. I'd even go so far to claim it might be easier to get commit access to a closed source product than open source.

As a youngling I was part of putting an easter egg into a rom-flashed hardware product by Very Large Company. There's no difference between being able to do that and something far more nefarious.

But from a legal point of view a corporation can act towards an employee or consultant in a way they cannot vs some anonymous open source contributor - and even though that doesn't help after the fact it's something that still feels reassuring to them.

in reply to daniel:// stenberg://

I have strong feels that it's not just open source that suffers from this.

At least with open source, one can look at the source.

in reply to daniel:// stenberg://

agreed, we can‘t detect all of them on code level. I‘m more wondering whether we could detect potentially dangerous patterns on organizational/structural level, e.g. low-change repos who suddenly come back to live, maintainer changes etc.
in reply to daniel:// stenberg://

in my mind, I imagine that there are probably some others in use right now...

and it is a LOT worst in closed source software, where no one can analise it thoroughly.
IOT products come to mind easily

in reply to daniel:// stenberg://

> But we are Open Source which allows everyone to dig, check, read code and investigate.

And *this* is the main difference between companies like Microsoft, Apple etc., and open source software on the other hand. In this case, you can see the exploit, you can find out who inserted it and you can check all of the history of the project, down to when those people first showed up.

If there's something like this in closed source, you wouldn't know. It would never be shown in public.

in reply to daniel:// stenberg://

a key thing people are missing here is that the backdoor was inserted into the #xz tarballs and not in the public git repo.
#xz
in reply to FobUpset

@FobUpset I'm not convinced that's strong factor seeing that he already managed to land quite a big portion of the attack in git.
in reply to daniel:// stenberg://

As much as I hate the current AI craze, I would be curious if there are any heuristics that we can learn from here (i.e. there were multiple identities involved that didn’t appear long before maintainers were pressured into accepting changes). Sentiment analysis is something we can do for some time now. But generally of course, I agree with you. There are no easy solutions.
in reply to daniel:// stenberg://

I guess the only way to fix something like this is to completely flip around how permissions work from the current denylist approach to an allowlist approach.

Something that, from my understanding, WASI is doing.

But the current model of doing things is so ingrained in our operating systems, programming languages, ABIs and, to some extent maybe even hardware, that it seems like an impossible thing to do retroactively.

in reply to daniel:// stenberg://

one way to reduce the risk would be to recompile the libraries as the distro gets built vs just using tarballs.
in reply to Thomas Lee ✅ :patreon:

@DoctorDNS I don't understand how this would help.

The attacker was the entire active ecosystem at the time. They chose to deliver the final payload as a source tarball instead of a git commit because they appear to have chosen to target distributions that were consuming the tarballs. But that was a decision of the attacking organization based on their targets' operational practices.

in reply to Stephen Bannasch

@stepheneb @DoctorDNS I'm aware. The message to which I replied didn't say "recompile from git sources"...

But even if it did. The other half of my post stands: the attackers will choose a model based on their target (here the distributors).

In this case, they assembled their attack using one git repository and one source code change in a "source archive". They could in a future attack distribute the components across multiple disparate components with seemingly unrelated maintainers.

in reply to Josh Soref (w/ screen reader)

@jsoref_using_a_screen_reader @stepheneb @DoctorDNS exactly. Starring too hard on the exact specifics used this time is pointless. The attack is of course crafted for the specific projects and their habits.
in reply to daniel:// stenberg://

Even then, this particular attack was carefully designed to get glossed over by code auditors!

I personally looked at XZ somewhat recently, and while I can't tell you whether I audited a malicious version...

And yes I was studying the tarballs!

This entry was edited (1 month ago)
in reply to daniel:// stenberg://

I think maybe the best thing that could be done is if a company really depends on a piece of software written as a hobby they should try to help the maintainer to avoid burnout. From what I understand of this whole thing, it all boils down to one developer suffering undue mental stress trying to keep up with the demands of users to the point of giving up, which the attacker thoroughly exploited. If we were all nicer to each other maybe this crap wouldn't be so easy.
in reply to daniel:// stenberg://

My favorite part of all of this is people going on about "resilience" and achieving it by building another version of everything (that also has to be compatible, otherwise how will people make use of it?):

"Then build in resilience. Defense in depth, and diversity — not a monoculture. OpenSSH will always be a target because it is so widespread, and the OpenBSD developers are doing great work and the target was upstream of them because of this. But we need a diverse ecosystem with multiple strong solutions, and as an organization you need second suppliers for critical software." (https://www.docker.com/blog/openssh-and-xz-liblzma/)

Does this mean we can expect another group to write a curl compatible library in Rust now so we have some resilience and diversity? (this is sarcasm, I feel some people might take this seriously. It is not serious, it is sarcasm).

in reply to kurtseifried (he/him)

@kurtseifried maintaining completely compatible duplicates has to be one of the more insane solutions to the problem that I have heard about!
in reply to daniel:// stenberg://

@kurtseifried yet that is exactly what is done in safety critical environments. During the Apollo program, there were two independent navigation computers, each developed by different teams (the primary one, the famous AGC was developed by a university, MIT I think? And the spare emergency computer was by IBM).

Completely different hardware and software, both of which could get the lunar lander back to the command module.

Perhaps this idea isn't so bad after all? At least for certain software.

in reply to Elias Mårtenson

@loke Those systems are comically simple and small compared to modern systems.

"Looking at transistor counts, the Apollo Guidance Computer had about 17,000 transistors in total in its ICs"

"The 1969 Apollo 11 mission (above) was the first to land men on the Moon. Since then, the most obvious advances have been in computing and electronics (especially in reducing size). The Apollo Guidance Computer had RAM of 4KB, a 32KB hard disk."

You can't do that with something like curl.

in reply to kurtseifried (he/him)

@kurtseifried just like the backup navigation computer couldn't do everything the AGC could, and didn't need to, no one uses all the features of curl.

You don't need to replicate all behaviour, only a small subset. These alternatives already exist, and if the interfaces were standardised, you'd be some way towards this already.

Of course, something like this could likely never be agreed upon by the community, but there are plenty of companies out there that should and could, contribute financially too such a project.

in reply to Elias Mårtenson

@loke @kurtseifried i will add that we got pretty good research on this in the 90s, i recommend to look at Leveson. Spoiler: it does not help, because they... Make the same mistakes.

There are also other dynamics here like the fact that the niche expertise necessary highly limit how many people that can work on it, which over time push toward convergence and monopoly.

in reply to Thomas Depierre

@Di4na @loke @kurtseifried yes, and these days we have open source which should ideally be a much better system to make a single implementation solid over time in ways doing two proprietary alternatives never could...
in reply to daniel:// stenberg://

It is difficult but the xz incident is also a success story: the backdoor was spotted before landing in stable Linux distributions.
#xz was probably chosen due to the presence of a corrupted xz file as part of the tests making it an ideal candidate for hiding data. In cryptography there are https://en.wikipedia.org/wiki/Nothing-up-my-sleeve_number - the same principle could be used to reject mysterious blobs from codebases. Yet many "bugdoors" can be introduced by creating subtle vulnerabilities and that's difficult to spot.
#xz
in reply to federico

yes, the xz backdoor was spotted, but we don't know how many others we have not spotted (yet). xz was in many ways an ideal target for this attack, but I imagine that there are many others as well, even if they would need slightly tweaked approaches
This entry was edited (1 month ago)
in reply to daniel:// stenberg://

sshd is a very high-value target, and it has a LOT of dependencies. It would probably be possible to reduce the risk by making it (or a replacement) simpler with fewer dependencies.
in reply to daniel:// stenberg://

Precompiled code would be the one thing that is likely to be seen again or in other existing attacks, I would like to see a GCC that puts specific magic in all object files and a linker or post install check that looks for it consistently.

An advanced attack can put the mark in new exploits but it gets harder to hide.

I guess this fits under dig, check, investigate.

in reply to daniel:// stenberg://

What we can do is at least fingerprint the methodology used by this same threat actor to discover if there were any other very similar attempts.

It's likely there will be common elements in how they reached out to various open source projects which were very common dependencies.

This won't catch all cases, but it could catch some.