The cyber-scare shows why the internet’s crowdsourced code is vulnerable
In 2020 xkcd, a popular online comic strip, published a cartoon depicting a teetering arrangement of blocks with the label: “all modern digital infrastructure”. Perched precariously at the bottom, holding everything up, was a lone, slender brick: “A project some random person in Nebraska has been thanklessly maintaining since 2003.” The illustration quickly became a cult classic among the technically minded, for it highlighted a harsh truth: the software at the heart of the internet is maintained not by giant corporations or sprawling bureaucracies but by a handful of earnest volunteers toiling in obscurity. A cyber-security scare in recent days shows how the result can be near-disaster.
On March 29th Andres Freund, an engineer at Microsoft, published a short detective story. In recent weeks he had noticed that ssh—a system to log on securely to another device over the internet—was running about 500 milliseconds more slowly than expected. Closer inspection revealed malicious code embedded deep inside xz Utils, some software designed to compress data used inside the Linux operating system, which runs on virtually all publicly accessible internet servers. Those servers ultimately undergird the internet, including vital financial and government services. The malicious code would have served as a “master key”, allowing attackers to steal encrypted data or plant other malware.
The most interesting part of the story is how it got there. xz Utils is open-source software, meaning that its code is public and can be inspected or modified by anyone. In 2022 Lasse Collin, the developer who maintained it, found that his “unpaid hobby project” was becoming more onerous amid long-term mental-health issues. A developer called Jia Tan, who had created an account the previous year, offered to help. For more than two years he, she or they contributed helpful code on hundreds of occasions, building up trust. In February they smuggled in the malware.
The significance of the attack is “huge”, says The Grugq, a pseudonymous independent security researcher who is widely read by cyber-security experts. “The backdoor is very peculiar in how it is implemented, but it is really clever stuff and very stealthy”—perhaps too stealthy, he suggests, because some of the steps taken in the code to hide its true purpose may have slowed it down and thus raised Mr Freund’s alarm. Jia Tan’s patience, supported by several other accounts who urged Mr Collin to pass the baton, hints at a sophisticated human-intelligence operation by a state agency, suggests The Grugq.
He suspects the svr, Russia’s foreign-intelligence service, which in 2019-20 also compromised SolarWinds Orion network-management software to gain extensive access to American government networks. Analysis by Rhea Karty and Simon Henniger, published on their Substack, suggests that Jia Tan made an effort to falsify their time zone but that they were probably two to three hours ahead of Greenwich Mean Time—suggesting they may have been in eastern Europe or western Russia—and avoided working on eastern European holidays. For now, however, the evidence is too weak to nail down a culprit.
The attack is perhaps the most ambitious “supply-chain” attack—one that exploits not a particular computer or device, but a piece of back-end software or hardware—in recent memory. It is also a stark illustration of the frailties of the internet and the crowdsourced code upon which it relies. For defenders of open-source software, Mr Freund’s eagle eyes are a vindication of its premise: code is open, can be inspected by anyone, and errors or deliberate backdoors will eventually be found through collective scrutiny.
In the shadows
Sceptics are less sure. Some code security and debugging tools did pick up the anomalies in xz Utils, but Mr Freund acknowledges “the number of coincidences that had to come together to find this”, including a series of technical but arbitrary choices he made while troubleshooting an unrelated problem. “Nobody else had raised concerns,” writes Kevin Beaumont, another cyber-security specialist. Software engineers are still probing the inner workings of the backdoor, attempting to understand its purpose and design. “The world owes Andres unlimited free beer,” concludes Mr Beaumont. “He just saved everybody’s arse in his spare time.”
The attack was detected and stopped before it could cause widespread damage. There is no way to tell whether Jia Tan, or the team apparently behind that persona, has been working on squirrelling into other vital pieces of internet software under other aliases. But security researchers are concerned that the foundations of the internet are ripe for similar campaigns. “The bottom line is that we have untold trillions of dollars riding on top of code developed by hobbyists,” notes Michal Zalewski, an expert. Other backdoors may yet lurk, undiscovered, elsewhere in the internet’s critical software.