XZ Utils Backdoor - A Supply Chain Attack

Deep Dive
A Bond Villan Level Supply Chain Attack

Did we just find our first cyber James Bond villain? First, let’s introduce our hero. On Friday, March 29, 2024, Andres Freund, an astute developer and now Internet hero, identified a backdoor in the xz-utils library. This open-source software is a command-line tool that manages fast data compression. It’s also open-source code that happens to be incorporated into many Linux operating systems.

As you read through this, remember that when dealing with open-source code, anyone can submit code changes, and if approved by the project owner or those with the right permissions, those code changes go into effect. These open-source projects are built into other applications and operating systems. This means that a small open-source project maintained by someone who built something for free can make its way into code millions use.

Now, on with our story...

Andres noticed that SSH logins took a long time and linked it back to the liblzma code, part of the xz package. As he dug further, he found malicious code deep in the code. It was obfuscated to make it difficult to find, and analyze. Although details are still limited, we know that the malicious code functions as a backdoor when certain criteria on the system are met. The known criteria include running a specific Linux distro (currently believed to be Debian or Red Hat-derived distributions), and the malicious versions of xz or liblzma must be present on the system.

The backdoor was simple but effective. It first hooks into the OpenSSH service, which is used for remote access, and loads a malicious payload. When the RSA_public_decrypt function is called (as it would during a key-based SSH login), the legitimate function is redirected to the backdoor code. Researchers observed this can be used to skip authentication during an SSH login. However, additional reports say it allows for remote code execution. Either way, it would allow an attacker to remotely access and interact with the system.

Even though the malicious code has been present since late February, the story of how it got there started three years ago! Evan Boehs created an interesting timeline for the user who committed the malicious code.

2021: Jia Tan created a GitHub account with the username JiaT75. Their first commit was minor, changing how an error message was printed in libarchive. There is speculation that this could have created a vulnerability, but it has since been patched.

2022: Lasse Collin, the creator of xz utils, exchanges messages with a few users about delays in making updates. In the exchanges, Jigar Kumar presses Lasse for not maintaining the code and urges him to get a new maintainer or co-maintainer. This is the first instance where Lasse comments that he has been working with Jia Tan and that they may play a bigger role in supporting xz utils in the future.

2023: JiaT75 merged their first commit to the xz utils project in January, meaning Lasse granted them elevated permissions to do so. Then, things get interesting…

1. JiaT75 updates contact information for Google’s oss-fuzz project from Lasse’s email to theirs. oss-fuzz is a tool that automatically checks for code issues and can spot some easy security issues.

2. JiaT75 also committed code that introduced the “ifunc” function. This allowed the backdoor to hook into the RSA_public_decrypt function…and was committed a full year before the full backdoor code was added.

3. JiaT75 then submitted code to oss-fuzz to disable “ifunc” for fuzzing builds. It all checked out because the contact information for oss-fuzz was previously updated to JiaT75.

2024: JiaT75 made the final commits that made the backdoor functional. Various online accounts then submitted update requests to major Linux distros to expedite getting the malicious xz-utils code into the main operating system code.

So how bad is this? It’s bad because this would allow a specific attacker (aka whoever Jia Tan was allegedly working with or for) to access certain Linux systems running the malicious code. The good news is that it was caught early enough that it wasn’t incorporated into many Linux distros, and those that it was were mainly pre-release. In all likelihood, we may have averted disaster here.

There is a larger question, though. This was a multi-year operation to build trust and gain the right permissions to insert malicious code into an open-source project that would lead to how SSH works and create an effective backdoor. It’s literally insane! And the fact that we got lucky to find this one begs the question of, what haven’t we found???

The right motivated attacker (like a nation-state) can spend the time and resources to conduct these attacks. With so many applications using open-source code, the question of how you validate all the supporting code your application relies on arises.

This is no small problem and may never reach an acceptable level of risk certainty. If that doesn’t sum up cybersecurity's challenges, I don’t know what does.

Spotlight
Not Your Ordinary Shipping Company

When you think of freight shipping, as one does, I bet cybersecurity and fraud don’t come to mind. And why would it? It’s so distant from the stories of ransomware and business email compromise we hear in the cybersecurity echo chambers every day.

And yet, fraudulent double brokering uses the same concepts of social engineering and impacts upwards of $700 million in freight annually. This is a significant problem for an industry with low margins.

So how does fraudulent double brokering work? I’m so glad you asked.

1. A broker (a middleman between shippers and truck carriers) assigns a load from a shipper to a freight carrier.

2. The fraudster creates a fake carrier or uses stolen information from a legitimate carrier to accept the load.

3. The fraudster then reposts the load on a load board with an inflated price that carriers would make it more attractive and get more carriers to bite.

4. The legitimate carrier picks up the load and transports it to the proper destination.

5. The fraudster invoices the broker and receives payment. Instead of then paying the legitimate carrier who delivered the load, the fraudster peaces out with the money like ALPHV’s exit scam.

6. The broker often covers the double payment to avoid embarrassment and preserve their relationship with the shipper.

The impact on brokers is eerily similar to that of businesses dealing with fraudulent funds transfers. They incur additional shipping fees from paying the invoice twice, potential legal fees, and increased insurance premiums from submitting a claim.

And, perhaps unironically, just like companies and their ability to defend against attacks, although 85% of survey respondents said they lost money due to the fraudulent double brokering in the prior quarter (yes, just the prior quarter to the survey), 77% of the respondents also felt confident they could detect the fraud. Something doesn’t line up there.

News
What Else is Happening?

🍎 Apple customers report MFA prompt bombing attacks related to Apple’s password reset feature. The attackers followed up with a phone call from the attacker impersonating Apple support, who attempted to get the user to supply the one-time passcode, which would initiate a password reset on the account and allow the attacker to take it over.

💧 Attackers are conducting a password-spraying attack against Cisco Remote Access VPN (RAVPN) services that operate on Cisco firewalls. These attacks just guess a large number of common passwords to log in.

📵 Researchers found that several free VPNs came with a catch. They turned your phone into a proxy that attackers could use to route traffic during cyber attacks, making it look like your phone was where the attack originated.

👕Another retailer dealt with the damage from credential-stuffing attacks. This time, Hot Topic (not to be confused with Hot Pocket) alerted customers that attackers gained access to their accounts using valid account credentials in November 2023.