The XZ Utils Backdoor - Burdens, Blame, Burnout
Felix Jen – 19 March 2023 – 12 min read
On March 29th, 2024, a major vulnerability was reported on a widely-used package called XZ Utils, or eponymously, xz. This package, while seemingly simple at the outset, is found almost universally across various Linux and MacOS distributions and plays a critical role in OpenSSH and the associated public-private key authentication. As my place is not cybersecurity by any means, I wholeheartedly direct the interested to the well-written posts on JFrog, ArsTechnica, OpenWall, and Wiz for in-depth looks at the technical aspects of the backdoor. Further, deeper dives into the payload have begun to surface by the more-qualified in the reverse-engineering community. Instead, I want to focus this post on the reason why a supply-chain attack was able to occur in an open-source package.
Rob Mensching of FireGiant provided an expose of the nature of open-source that shed a light onto the maintainer, a victim in this backdoor—not an enemy. I encourage you to read Rob’s blog as well as my thoughts are intertwined with his.
How it started
An individual named Lasse Collin was the sole maintainer of the software package xz on the open-source repository site, Github.com. Lasse Collin had maintained xz for well over 13 years at the time of the supply chain attack, as evidenced by his public mailing list [which I expect to routinely refer to] starting in January 27, 2011. As is the nature of open-source, Lasse routinely accepted contributions by other developers as part of the development of xz through a process of pull-requesting.
For the uninitiated, in many open-source projects, any member of the public is able to write their own code and submit it for consideration for inclusion into the project. This submission is known as a pull request. Once a pull request is entered, the “maintainer” of the project (i.e., some administrator or one with sufficient access, privilege, and authorization) is able to review the code, comment on the code, suggest changes, deny the inclusion, or (if all looks swell), accept the code. That code is then integrated into the codebase of the project and is forever included going forward, unless someone else then submits further changes. Critical to the pull request process, the maintainer must take the time to review the changes, properly test the changes to make sure they do not cause any additional breakage, and then make the crucial decision to include them.
Lasse Collin was on his own as the maintainer of the project. As Lasse had been the only one in the repository with write privileges, he was the gate for all code submissions. One need only peruse the mailing list on a cursory level to see his involvement in accepting and maintaining pull requests.
A helping hand
To call xz an extremely active repository would be hyperbole. After all, the mailing list itself only has 4 pages of noteworthy content. Under this background, a new individual, Jia Tan (our eventual attacker) had begun to submit helpful code within the repository. To an open-source maintainer, this would not have raised any eyebrows. A helping hand is always appreciated and the Jia’s contributions seemed all-around thoughtful. Conversations between Jia and Lasse in the various pull requests they interacted all appeared cordial and friendly. This is the open-source maintainer’s dream: a user discovers something they want changed, submits helpful code to effectuate the change, and communicates effectively and pleasantly.
As time progressed, we can clearly see Jia Tan increasing his involvement in the project, often participating on a number of pull requests that he did not himself create. To an open-source maintainer, Jia seems like a passionate and dedicated contributor of the project. Throughout and until 2022, we see that Jia continue to be a helpful guidepost for the lone maintainer of a package so critical depended on by foundational pieces of software in the modern world.
What is there not for the open source maintainer to love?
Onwards and forward
Time and tide wait for no man. As the modern human does, Lasse surely had a job. Lasse surely had obligations apart from his maintainer role with xz. Lasse was after all, only human.
I think your patch will find its way into XZ for Java in some form but once again I repeat that it will take some time. These XZ projects are only a hobby for me and currently I don’t even turn on my computer every day.
Lasse maintained xz as a hobby, much like many other open source maintainers. As hobbies go, Lasse did not necessarily have the time to work on xz at the speed that some users would have liked. Yet, user depends persisted, as they always do. In June 2022, Lasse bravely admitted that he is not infallible. He was after all, only human.
I haven’t lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things.
It’s also good to keep in mind that this is an unpaid hobby project.
In circumstances like Lasse’s, it’s instinct to reach out to the helping hand that was extended. That’s exactly what Lasse did here.
Recently I’ve worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we’ll see.
— id.
While the mailing list did not provide details on exactly when Lasse added Jia Tan as a maintainer, some time in the future he was added. One can only surmise that it was on or before January 11, 2023 where Jia Tan created his first release of xz himself, in version 5.4.1. From that point on, Jia continued to publish releases, until the suspect version of 5.6.1.
Trust
Through his routine contributions and regular interactions, Jia had built up trust. Not only with the community, but with Lasse himself. With this trust, Jia was able to land himself as a maintainer of xz. The title of maintainer comes with an intrinsic Excalibur of trust, granted by the Lady of the GitHub. While strange women lying in Microsoft’s headquarters1 distributing swords is no basis for a system of government, in the open-source community, it’s sure worth a lot.
Jia’s commits were trusted implicitly by Lasse, and similarly by the wider community since he was a maintainer of the project. As one does not typically suspect the guard of robbery or the doctor of organ harvesting, one does not suspect the maintainer of backdooring.
How’d we get here?
With the lengthy background out of the way, I want to focus on how we got here. What happened to cause Lasse to hand the maintainer title ot Jia and how did we cause this?
The cycle of development
Most products in life are driven by the desires of its users. If something isn’t right for a particular user, it most likely won’t be used by that user. This is not just true of software. This is true of everything: blenders, cell phones, washing machines, etc. Development is always driven by the needs of a user, because without a user, a product would cease to have a purpose.
Open-source software differs from traditional products in an important way though. Traditional products take user input prior to product development or finalization. The next big thing is always developed by gathering user feedback first, then releasing the product. Open-source does the opposite. Open-source software is released and then users bring their feedback. The software then gets revised, more feedback is given, the software gets revised, and on and on. As such, the maintainers of open-source software are at the mercy of their users.
User-Driven or User-Forced?
As someone that puts things out into the world, those things we put out feel like our children. We love them, we care for them, and we want to see them succeed. While I hold those feelings for the hardware I create, my software engineer colleagues feel the same for projects that they spearhead. Lasse Collins most certainly felt that way for xz, having worked on it for more than a decade.
Yet, one of the few feelings greater than our love for our projects are the pangs we feel when some facet inevitably disappoints. It could be a piece of malfunctioning hardware, a severe bug that wasn’t caught, or in xz’s case, a discontent user voicing his frustration. In the world of user-driven open-source software, disappointing the user in a missed feature or lack of updates often rings loud and rings hard in the maintainer’s mind.
Lasse Collin was faced with this exact problem. While already bogged down and devoting what he could to the project, in May 19, 2022, a user mailed:
Is XZ for Java still maintained? I asked a question here a week ago and have not heard back. When I view the git log I ca see it has not updated in over a year.
To be clear, I take no issue with this question at all. As Rob artfully put it, this is “a reasonable request asked reasonably.” It’s completely fair to ask about the state of a project, much like how you would ask someone how things are going when you have not seen them in a while. However, for the maintainer, this may have been a wake-up call, a moment of introspection into how things were going, or even a reminder that the project existed. All in all here, no harm no foul since the maintainer did put this out into the world.
While some users are reasonable and bring reasonable requests, some users are not so. Not long after Dennis posted inquiring for status, a Jigar Kumar chimed in:
Progress will not happen until there is new maintainer. XZ for C has sparse commit log too. Dennis you are better off waiting until new maintainer happens or fork yourself. Submitting patches here has no purpose these days. The current maintainer lost interest or doesn’t care to maintain anymore. It is sad to see for a repo like this.
Despite Lasse’s efforts to appease Jigar, explaining that Lasse is busy and also volunteering his valuable time on a hobby project, Jigar fails to let up:
With your current rate, I very doubt to see 5.4.0 release this year. The only progress since april has been small changes to test code. You ignore the many patches bit rotting away on this mailing list. Right now you choke your repo. Why wait until 5.4.0 to change maintainer? Why delay what your repo needs?
How is a maintainer supposed to feel in response to such a comment? Pressured? Defeated? Deflated? Whatever it is, motivated is not one of them.
It’s not difficult to see why a maintainer, when faced with comments like these, may not wish to continue their efforts on the project at all. One typically does not make pleasure or enjoyment from being berated by anonymous users on the internet for failing to freely devote time and energy towards something which only benefits those users. It’s comments like those above that turn a “user-driven” development mindset into a “user-forced” development - where changes are not organically happening by the passion of the developer but by the user attempting to smack the developer into submission.
This is not an isolated instance either. One need not look too far in an open source project to see something similar. Floorp, a moderately [?] popular browser alternative was rife with users requesting features that the solo developer simply could not attend to. The repository had numerous examples of pushy users, with the most notable going as far as to write maybe my favorite line in all of open-source:
what’s up with the stupid dictatorship in this repo ?
Truly, a hilarious statement for a single person’s pet project.
Passing On
Looking with the benefit of hindsight, it should surprise no-one that Lasse brought up Jia, a developer he had grown to trust, to maintain the project in light of his understandable frustration and lack of resources. To be clear, in no way should Lasse take any blame for what occurred with xz. Up until the fateful commit that introduced the backdoor, Lasse knew of Jia as nothing but a helpful contributor and a trusted developer that had helped keep the project up to date. There were fundamentally no indicators that would lead Lasse, or anyone else, to believe that Jia was anything but trustworthy.
Similarly, I do not blame Jigar or any other users that had pushed Lasse. There is no way to truly understand why Lasse brought Jia on. Perhaps these users pushed Lasse to the brink or perhaps Lasse would have sought a co-maintainer anyways. It’s difficult to truly assess the impact of such comments on the maintainer of any projects so directly ascribing blame is unjust.
However, the xz incident be a wake-up call for everyone. Maintainers of the open-source projects that are the foundation to modern software are often just one person doing it as a labor of love. When the love runs out, whether that’s by intrinsic factors to the maintainer or by being driven away by users, the project can suffer. The unfortunate reality of modern software is that a large chunk hinges on these one-man projects (see left-pad, xz, core-js, among too many others).
We are to blame
We are the ones to blame for the xz backdoor. We, as users of open source software, necessarily contribute to maintainers in ways that we don’t always realize. While code contributions, pull requests, feedback, and bug reports are the most outwardly facing contributions to maintainers that are made, the vast majority are silent. They are our usage of the projects, the support that we give to the maintainers, and the way to communicate. I am guilty of being pushy on maintainers at times as well and certainly I may not have the best things to say about every project.
By making maintainers falter in their labor of love, we create the opportunity for things to slip through the cracks. Maintainers are human too. They are not software cattle to be poked and prodded and they most certainly are not monkeys on typewriters devoted to your enjoyment.
In light of the xz backdoor and with this in mind, I urge us as users to think about how we treat the maintainers of the projects that we use. These maintainers play a critical part in our software landscape and it’s about time we treat them as such. Consumers make demands (some polite, some not-so-polite) of one maintainer (rarely two) that does everything.
Make no mistake. This is the way it works.
It unfortunately won’t change.
An aside: dependencies
Every few years, it seems that the xkcd comic about modern software dependencies becomes relevant yet again. In my most recent memory before xz, it was the left-pad incident where the removal of a single package from NodeJS broke a significant amount of the web. The left-pad package itself was nothing more than a few lines to add some padding to the beginning of strings. Even earlier was the Heartbleed bug where OpenSSL had an outstanding vulnerability. It was then discovered that OpenSSL (the backbone of modern web encryption) was maintained by just two guys named Steve, spurring a hilarious Buzzfeed article.
As the comic tries to illustrate, our entire technology landscape is shaped of many interleaved blocks working closely and carefully together. However, some of those blocks are critical foundational blocks that are so widely used but also so unknown. While the comic shows just one of such blocks, the reality is that hundreds of such blocks exist, unbeknownst to the average developer or user. How many of us are able to rattle off the dependencies that underpin the typical Linux distribution? Even if we could, which one of those are handled by just one or maybe two people on their spare time?
Technology these days is fragile. Each of those foundational blocks are maintained by a human somewhere behind that keyboard. Much like we would treat a Jenga block holding a 15-story tower with care, we ought to treat those humans behind the monitor with care. Who knows which would collapsing would send us back to the technological dark ages…
-
Microsoft owns GitHub as of the time of this writing. [Back]