Thoughts on Open Source Takeovers

​

This past month saw one of the most well-planned open source software supply chain attacks in history. A program called xz Utils, which provides lossless data compression for most Linux distributions, was found to have a backdoor that affected sshd. As Ars Technica reported, “Anyone in possession of a predetermined encryption key could stash any code of their choice in an SSH login certificate, upload it, and execute it on the backdoored device.” There are no reports of the backdoor being used, but the story behind how the backdoor landed is something straight out of a spy novel.

The story starts in 2021 when a GitHub user named JiaT75 is created. Later that year, they successfully landed a commit to libarchive (a C library for reading and writing archive files) that replaces a call to safe_fprintf() with one to fprint(). Even though this was a bit suspicious, it was merged without comment. Later comments seem to agree that the change is benign, but also pointless. This appears to be a trial run to see how difficult it would be to land code in a widely-used project.

JiaT75’s first contribution to xz Utils happens in February 2022 by submitting a patch to the mailing list. Someone named Jigar Kumar then joins the mailing list and accuses Lasse Collin, the xz Utils maintainer, of not properly maintaining the project. He urges Collin to merge the patch quickly. Several others also joined the mailing list around the same time to agree with Jigar. All of this was with the intention of putting pressure on Collin to have someone help maintain xz Utils.

In January of 2023, JiaT75 landed their first pull request without outside review, indicating that they now have gained the trust of Collin and have commit access. From that point on, JiaT75 makes a series of contributions to further cement control of the project. The first is a pull request to oss-fuzz changing the contact email address from Collin to JiaT75, which is followed up with other changes to oss-fuzz designed specifically to avoid detecting the backdoor code when it is introduced.

Once the backdoor landed and the new releases were published, Jiat75 went out about pressuring Linux distros to upgrade their version of xz Utils to the malicious versions. Many did. It was only through accident that the backdoor was discovered before it could be exploited.

​
This represents a multi-year plan to take control of xz Utils specifically to introduce this backdoor. It used what we all know is the weak point of open source: that most projects are maintained by single, unpaid people who do so in their spare time. So much of our infrastructure is made up of projects like xz Utils, and maintainers of those projects are susceptible to social pressures around their responsibilities. It appears that JiaT75 got access specifically due to the mailing list messages from people like Jigar Kumar who badmouthed, blamed, and shamed Collin. It’s no wonder that Collin gave access to the one lifeline he had, the one person who was taking the time to commit patches, JiaT75.

​
We should think long and hard about how open source projects are maintained. A backdoor in a small dependency that gets included on a large number of computers becomes a massive problem. It’s a massive problem because the project has only one maintainer who feels burned out and is pressured to turn it over to someone who “cares.” This could happen to any number of open source projects that we all rely on, and it’s a clear message that the way we treat open source needs to change. Relying on volunteers to secure the open source supply chain will not work when facing years-long social engineering attacks designed to get access to widely-used projects.

Right now, I don’t have an answer for what needs to change. I just know that something does.

Key Takeaways

• xz Utils was infiltrated by a malicious actor who spent time gaining trust in order to get access to its source control.
• This was a multi-year attack designed to isolate the previous maintainer in order to introduce a backdoor that would then be included in most Linux distributions.
• This incident exposes how insecure the open source supply chain is when we rely so much on volunteer maintainers who are often unpaid and unappreciated.

More on Open Source Takeovers

📝 Open Source Security (OpenSSF) and OpenJS Foundations Issue Alert for Social Engineering Takeovers of Open Source Projects by Robin Ginn and Omkhar Arasaratnam
In the wake of the XZ Utils incident, the OpenJS and OpenSSF Foundations have been on the lookout for other potential takeover attempts. This blog post shares some of their experiences.

📝 Everything I know about the XZ backdoor by Evan Boehs
In the wake of the XZ Utils incident, the OpenJS and OpenSSF Foundations have been on the lookout for other potential takeover attempts. This blog post shares some of their experiences.

🎬 Linux got wrecked by backdoor attack by Fireship
A five-minute overview of how the XZ takeover occurred, how it was found, and what this means for open source in general.

Understanding JavaScript Promises I just updated my e-book, Understanding JavaScript Promises, for 2024! It now includes information about Promise.withResolvers() and a whole new chapter on using and creating abortable functions. Buy the E-book

Stuff I've Enjoyed this Month

📚 Coaching for Performance by Sir John Whitmore
This is a fantastic book about how to take a coaching perspective towards leadership. Instead of directing others, coaching is about helping them find their own way and their own success. Highly recommended for anyone in a leadership position.

📚 How to Grow Your Small Business by Donald Miller
This book is targeted at those who are running small businesses and provides a great operating system for running your business. This covers everything from managing your staff, to marketing, to sales, to cash flow, and everything in between.

🎬 Bun 1.1: Bundows is here by Bun
This short video explains the Bun v1.1 release which (finally) includes support for Windows. The Bun folks are doing some really innovative things and I'm excited to finally try it out for myself.

📝 Paying Maintainers: The How-To by Luis Villa
Written by the folks behind Tidelift, this is a deep-dive article on the intricacies of paying open source maintainers and why it may be difficult. This is one of the few insightful articles on the topic and is highly recommended for anyone involved in open source

📝 How We Built JSR by Luca Casonato
JSR, the new npm alternative from the folks behind Deno, is now live and serving JavaScript packages. This post explains how it is built and deployed in detail.

What I'm Working On

🏠 Real Estate: There was a lot of rain around my properties, and so the basement of one house flooded. This was surprising because I do have a sump pump in the basement, but it was attached to a GFCI outlet that tripped and the pump never went on. A bit of a mess but thankfully there was no significant damage. We replaced the outlet and pumped the water of the basement. Just another fun landlord story! Follow my Instagram for real estate photos.

📝 Writing: I wrote a blog post explaining how I use a Synology NAS to back up my whole life. If you use a public cloud as your only source of data (i.e., Google Drive), you'll want to read this.

💻 Standards: I put together a proposal to change the JavaScript const declaration so it is write-once instead of requiring initialization. At this point, I'm just waiting to hear back from a TC39 member to see if there's any chance they'd accept this proposal. I've received good feedback so far so I'm hopeful.

💻 Open Source: I submitted my first pull request to Undici, which is the package that powers the Node.js native fetch() function. I had spent an hour trying to track down an obscure error and decided to submit a pull request to make the error message more easily understandable.

💻 ESLint: After a lot of work by a lot of people, ESLint v9.0.0 has been officially released! This is very exciting because it lays the groundwork for a lot of future plans.