Here's a thorough analysis of all the commits by "Jia Tan" from 2023-08 through 2024-03, showing the many legitimate code changes done before the introduction of the #xz #backdoor:

https://tukaani.org/xz-backdoor/review.html

Excellent summary by Solar Designer on oss-security of what's happened in the last two weeks in response to the #xz #backdoor:

https://www.openwall.com/lists/oss-security/2024/04/16/5

Noteworthy:
- #OpenSSH implemented systemd notification
- #systemd moves to dlopen(3) for some dependencies
- another detailed timeline at https://research.swtch.com/xz-timeline
- similar social engineering takeover attempts suspected in #OpenJS and #OpenSSF

At Sovereign Tech Fund, we're following the #xz incident closely and listening to the many voices in the #FOSS maintainer community.

What's clear to us is that the xz incident shows the need for structural change:

https://www.sovereigntechfund.de/news/xz-structural-change

xz incident shows the need for structural change

^{Sovereign Tech Fund}

I am getting tired of reading about the #xz #security issue as if it is all about issues within #opensource. It is much bigger than that, and those takes conflate the problem with the solution.

So I wrote "The xz issue isn't about Open Source" here: https://changelog.complete.org/archives/10642-the-xz-issue-isnt-about-open-source

The xz Issue Isn’t About Open Source

You’ve probably heard of the recent backdoor in xz. There have been a lot of takes on this, most of them boiling down to some version of: The problem here is with Open Source Software. I want…

^{The Changelog}

"You have to understand, we’re responsible for taxpayer money here. We can’t just make a donation to your open source project."

— a national government who relies on #Matrix when being asked to support it financially

Read more about the problem and some initiatives that are responding to it:

https://matrix.org/blog/2024/04/open-source-publicly-funded-service/

#FreeSoftware #OpenSource #FOSS #FLOSS #funding #xz #sustainability

Open Source Infrastructure must be a publicly funded service.

Matrix, the open protocol for secure decentralised communications

^{Matthew Hodgson (matrix.org)}

Open source infrastructure *must* be a publicly funded service, and funders need to support maintenance – not just new feature development 📣

This is on our minds this week in the wake of the #xz news, and as we continue to seek funding to support #Matrix.

Read the latest from project lead, @matthew: https://matrix.org/blog/2024/04/open-source-publicly-funded-service/

#OpenSource #FOSS #OpenStandards

Open Source Infrastructure must be a publicly funded service.

Matrix, the open protocol for secure decentralised communications

^{Matthew Hodgson (matrix.org)}

"What it means is that there is no supply chain here. Because there is no supplier. I am not providing you something that you bought from me. There is no relationship. I put something online because I wanted to. The fact you made your product depend on it is your responsibility. Not mine. Not the one of the providers. We provide libraries. We do not supply them. You cannot apply rules to me. […] So all your Software Supply Chain ideas? You are not buying from a supplier, you are a raccoon digging through dumpsters for free code. So I would advise you to put these rules in the same dumpster."

🔥🔥🔥

https://www.softwaremaxims.com/blog/not-a-supplier

#xz

Any experienced C developers among my followers? #BoostsWelcome.

Expat, arguably the world's most popular #XML parser, is understaffed and without funding. As #xz has shown, situations like this are dangerous.

Last month, maintainer Sebastian Pipping put up a plea for help at https://github.com/libexpat/libexpat/blob/R_2_6_2/expat/Changes

(I would help myself, but my C skills barely surpass "Hello, World".)

Found via @timbray - https://cosocial.ca/@timbray/112203547801373427

#libexpat
#SoftwareSupplyChainSecurity #OpenSource #OpenSourceMaintainer
#C

libexpat/expat/Changes at R_2_6_2 · libexpat/libexpat

:herb: Fast streaming XML parser written in C99 with >90% test coverage; moved from SourceForge to GitHub - libexpat/libexpat

^GitHub

Tim Bray

2024-04-02 20:37:47

I think the #xz incident is teaching us that our infrastructure is dangerously fragile in the face of well-organized/funded attackers. The response isn’t “try harder” or “donate to your OSS project”, it needs to be institutional, professional, and at scale.

So, here’s my proposal, called “OSQI”, aimed at starting a how-to discussion: https://www.tbray.org/ongoing/When/202x/2024/04/01/OSQI

OSQI

^{ongoing by Tim Bray}

Jia Tan's history of commits on #xz suggests that every png file in gcc or apache or whatever is a possible attack vector now.

https://joeyh.name/blog/entry/reflections_on_distrusting_xz/

I think the #xz incident is teaching us that our infrastructure is dangerously fragile in the face of well-organized/funded attackers. The response isn’t “try harder” or “donate to your OSS project”, it needs to be institutional, professional, and at scale.

So, here’s my proposal, called “OSQI”, aimed at starting a how-to discussion: https://www.tbray.org/ongoing/When/202x/2024/04/01/OSQI

OSQI

^{ongoing by Tim Bray}

While the #xz backdoor has everyone focusing on ways to make and keep open source sustainable, let's talk about the systemic abuse reinforcement mechanism that is the CVE database. Case in point: CVE-2023-45853.

This is a "vulnerability" that is reported for an _example_ source code file included in the zlib package. NIST has inexplicably classified this as a 9.8 out of 10. They fail to attribute the report: https://nvd.nist.gov/vuln/detail/CVE-2023-45853

Its also kinda enlightening on how distros react to the #xz backdoor:
* #arch "lets rerelease the version from the untrusted party, we run autogen.sh ourselves now"
* #debian "lets roll back to the last version not having any changes by the untrusted party and rebuild our infra from scratch"

I know which of these I trust more as an upstream ...

There simply is no established or easy way to detect backdoors done the #xz way. We give powers and trust to maintainers because that is the development model.

Anyone suggesting there is an easy fix has not understood the issues at hand.

But we are Open Source which allows everyone to dig, check, read code and investigate.

Stuff I already wrote that other people might be open to reading this week, because of the #xz incident:

https://www.harihareswara.net/posts/2021/sidestepping-the-pr-bottleneck-four-non-dev-ways-to-support-your-upstreams/ Four Non-Dev Ways To Support Your Upstreams (Pass this along to executives who are asking "how can we prevent this in our dependencies?")

https://www.harihareswara.net/posts/2023/user-support-equanimity-potential-cross-project-tools-practices-open-source/ Potential cross-project #opensource tools and practices that you/we can implement to help lighten the load on each other

1/n

I think this #xz thing is gonna go down as the day #FLOSS officially lost its innocence.

You know what I hope comes out of this moment of sincere sadness for those who care about this stuff?

A sense that we will no longer be abused by Megacorps who build on the backs of our work but can't be arsed to help fund that work despite the fact that literally THE CONTINUED EXISTENCE OF THEIR BUSINESS depends on it.

Do you think #google, #amazon or #facebook could exist in their current form without oodles of super high quality free software to run their server swarms on?

Because they can't. For those of us old enough to remember, just imagine "the cloud" if every virtual server required a Solaris or SCO license.

If you've given all you can give, walk away. Open source is wonderful and amazing but you are human and your health and well being is more important. I don't care what falls down as a result.

Three years ago, #FDroid had a similar kind of attempt as the #xz #backdoor. A new contributor submitted a merge request to improve the search, which was oft requested but the maintainers hadn't found time to work on. There was also pressure from other random accounts to merge it. In the end, it became clear that it added a #SQLinjection #vuln. In this case, we managed to catch it before it was merged. Since similar tactics were used, I think its relevant now

https://gitlab.com/fdroid/fdroidclient/-/merge_requests/889

Search improvements: Sort based on keyword matching and removed alphabetic sort (!889) · Merge requests · F-Droid / Client · GitLab

The search results are pretty unusable currently. So I've changed it to show apps in this order: App name matches keyword, summary matches keyword, description matches keyword. Also,...

^GitLab

@kurtseifried and I recorded an #osspodcast episode about the #xz saga last night.

As much as I wish this was an April 1 joke, it's not

We try to break things down as simple as we can, and we end the show on a high note. There's no reason to lose hope, but there's a lot of hard work to do if we want to fix this.

Good luck to you all this week, it's going to be a wild ride

https://opensourcesecurity.io/2024/04/01/xz-bonus-spectacular-episode/

XZ Bonus Spectacular Episode

Josh and Kurt talk about the recent events around XZ. It’s only been a few days, and it’s amazing what we already know. We explain a lot of the basics we currently know with the attitud…

^{Open Source Security}

Put yourself in Jia Tan's shoes, the malicious contributor to the xz backdoor...

It's been, what, two... three?... years since you started this campaign. You've had the entire support of your team and of your chain of command.

Your coders created a complex and sublime backdoor. A secure! backdoor that only you and your team could connect to. Heck it can even be deleted remotely. This is clean code. A responsible hack that doesn't open up the backdoor for others to hijack.

You spend years on your long con - your social engineering skills are at the top of the game. You've ingratiated yourself painstakingly into multiple teams. Finally it all pays off and you're ready to go!

You succeed multiple times in getting your backdoor inserted in all the major Linux distributions!!! Now its just a matter of weeks before it makes it to production and stable releases!

This is the culmination of years of labor and planning and of a massive team and budget.

You did good.

This will get you promoted. Esteemed by your colleagues and leadership alike. Your spouse and kids will understsnd why you haven't been at home lately and why you've spent all those late nights at the office.

It's finally going to pay off.

But what's this?! Some rando poking around in their box running a pre-release unstable version of linux has found everything?!?! It's all being ripped down?! And on a Friday before a western holiday weekend?!?!

Fuck. Fuck. FUCK!!!

Three years for nothing!!! My wife is going to leave me! I missed my kid's recital for this!!! They'll hate me because I told them it was worth it. Daddy will be able to play with you again once Daddy finishes this last bit of work. But it was all for nothing!!!

Leadership took a big risk on me and my team but I kept assuring them it would pay off!

It would be one thing if another nation state found it and stopped it. But one random dude poking his nose where it shouldn't belong?! Ohhh fuck, I'm going to be fired. We're going to lose our budget. My team is going to be fired. I've let down everyone that ever believed in me and supported me and relied on me!

Oh fuck!!!

#xz #backdoor #xzBackDoor #cve #cve20243094 #infosec #hacking #FOSS

The things I don't like about the discussion on whether this is a state actor behind the #xz backdoor are:

* It doesn't change the response for pretty much anyone except a narrow group of professionals. Ultimately I don't know that it matters for most of us if this was a state attacker or some kid who wants a way to get op privileges.

* It distracts from next steps.

* Would they think that if the actor were named John? Will this increase suspicion of anyone with a "foreign" sounding name?

People are afraid of running unaudited `curl | sh`, but nobody bats an eye on 24707 lines of obfuscated garbage in `./configure`.

#xz

🤯 The level of sophistication of the XZ attack is very impressive! I tried to make sense of the analysis in a single page (which was quite complicated)!

I hope it helps to make sense of the information out there. Please treat the information "as is" while the analysis progresses! 🧐 #infosec #xz

Hey funders,

You know you could just... give... the money... to projects that need it. Like software libraries that ARE IN EVERYTHING.

No grants. Don't make tech nerds write grants.
Don't make the tech nerds hire grant nerds to write grants.

FFS don't fund research into this problem with a budget of double what it would take to SOLVE THE PROBLEM for a significant number of open source projects with code that is, again, IN EVERYTHING.

#xz

Regarding the #xz backdoor, I've seen statements like "if you're not running a publicly exposed sshd, you're safe". This is not the case and reflects a pretty outdated security mindset. You're still vulnerable, because you shouldn't assume internal connections are inherently trustworthy.

Yes, it limits exposure, but that's not the same - you still have a high-severit incident on your hands. Anyway, just here stating the obvious, as usual. ✌️

thinking about the #xz backdoor.

it might have been the maintainer, a long term plan to backdoor after training trust.

it might have been someone else getting access to the maintainer's credentials.

it was discovered almost by accident, because xz's used in many places.

investigations found the backdoor in the code.

would we have discovered this in something closed source? maybe the performance degradation, but we wouldn't have seen the code then...

So, Philipp Kern dropped by asking if we could do some #ReproducibleBuilds verifications of recent Debian Security updates, given, well the whole #xz mess... and that our build infrastructure may have run compromised code at some point...

So I did a quick pass at a handful of updates and everything verified ok so far, though I skipped some of the probably more juicy targets such as chromium and firefox:

https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/003321.html

Debian is reproducible enough to at least try this sort of thing!

https://boehs.org/node/everything-i-know-about-the-xz-backdoor

I have begun a post explaining this situation in a more detailed writeup. This is updating in realtime, and there is a lot still missing.

#security #xz #linux

Content warning: quick clarification on the xz backdoor thing Click to open/close