Cloudflare's Impressive Thanksgiving Investigation

Spotlight
Are We Sitting on a Cyber Tinderbox?

Perhaps you share a similar sentiment as me. I get disgusted when I hear about cyber scams that prey on innocent people. Especially those that are more vulnerable than others.

Things like phantom hacker scams that target senior citizens, attempting to drain their life savings by liquidating their assets and tricking them into handing it over to a stranger. All the while, the victims think that they are protecting themselves.

And yet, those scams are just the start of the depths cybercriminals will go. A recent article from Brian Krebs highlighted a significant law enforcement win in which the FBI arrested Noah Urban, who was linked to the notorious 0ktapus gang that wreaked havoc on tech companies.

Krebs went into more detail on one of this criminal’s favorite tactics. SIM-swapping, in which they transfer the phone number of their target to a device they control. This allows them to control SMS MFA prompts. I have a full video on MFA bypass attacks, including SIM-swapping, in this YouTube video.

While the arrest is a win for law enforcement, something else stood Krebs shed light deeper into the criminal aspect of SIM-swapping and, like gangs, how far these cybercriminals are willing to go to protect their market share and interests.

Krebs covers one “holder,” someone who physically holds the devices the phone numbers are transferred to, who was abducted, beaten, and held for a $200K ransom. Supposedly, that person was shot in the leg during the ordeal but survived.

Why did this happen? Because the SIM swapper who hired the holder ordered it after the holder withheld some of the stolen funds collected during a cyber attack. Quite the escalation if you ask me.

That brings us to the tinderbox question. Krebs highlights in another blog post the concept of “violence-as-a-service.” This is exactly what it sounds like. Anonymous Internet users can hire people in the physical world to carry out a variety of violent attacks. Things like:

• Throwing a brick through someone’s window (getting “bricked”)

• Firing weapons into houses in the middle of the night

• Driving a car through someone’s window

Cybercriminals are becoming more connected to people willing to carry out violent tasks. Or even using law enforcement in swatting attacks, which I covered in last week’s newsletter. This makes me question…are we standing on a cybercrime tinderbox?

As one cybercriminal escalates, another responds with a bit more force. It may happen slowly over time, but violence begets violence. I only hope ransomware actors don’t try to adopt these techniques in an attempt to gain more leverage.

Deep Dive
Cloudflare Had a Different Thanksgiving

You might recall the Okta breach back in October, which I covered in a prior newsletter. We knew from that incident that several companies were impacted as a result. One of which was Cloudflare.

What we (and Cloudflare) didn’t know was that even though they remediated the incident, they missed something. And, of course, the attackers found what was missed and took advantage of it.

Let’s dig into what happened, which they cover in greater detail in their own blog post.

Root Cause: Okta’s October compromise led to the theft of Cloudflare session cookies and credentials. Cloudflare detected unauthorized activity in their environment at that point and took action to kick the attacker out.

While Cloudflare went to work on resetting and rotating all the impacted credentials, there was a slight mishap. One service token and three service accounts were not rotated as they were “because mistakenly it was believed they were unused.”

Before you judge, remember these things happen, and we’re talking about an advanced threat actor who is willing to spend ridiculous resources to hack into their targets.

Initial Access: The attacker discovered that those credentials were still valid and began probing Cloudflare’s systems on November 14, 2023, to see where they were valid. Ultimately, they successfully accessed Cloudflare’s Atlassian JIRA and Confluence systems on November 15, 2023. This was possible by chaining two of the three compromised but unrotated credentials to bypass multiple authentication points.

On November 16th, the attacker searched through Confluence and JIRA, looking for information on remote access, secrets, vulnerability management, network access, and MFA bypass. This included accessing 36 JIRA tickets and 202 wiki pages…suggesting they enjoy reading a lot of dense material to further their mission…the dedication is real. The same day, the attacker created a new Atlassian account for persistent access to the Atlassian environment.

Lateral Movement: After digesting the stolen information, the attacker returned on November 22, 2023. This time, they brought a tool. Specifically, the Sliver Adversary Emulation Framework, which is basically an attacker’s toolbag to facilitate remote access, escalating privileges, lateral movement, and all the other fun things attackers do once they gain access to your environment.

The attacker used this to poke around other areas of Cloudflare’s environment. They accessed a non-production console server in a new data center in Sao Paulo, Brazil, and began poking at various code repositories. They viewed 120 code repositories, 76 of which were related to how backups work, how the global network is configured/managed, how identities work at Cloudflare, and Cloudflare’s use of Terraform and Kubernetes tech.

The Discovery and Response: On November 23, 2023, the attacker was busted after adding one of the compromised service accounts to an administrator group. This set off alarms for Cloudflare’s SOC, which was alerted within two minutes to the activity. Within 12 minutes, the SOC began investigating what was going on.

Within 20 hours of discovery, Cloudflare investigated, contained, and eradicated the attacker from the environment. The last known attacker activity occurred on November 24.

But Cloudflare didn’t stop there…

“Code Red” Remediation and Hardening Effort: Cloudflare was spooked by the access. What followed was one of the most impressive remediation efforts I’ve seen in over ten years of incident response. Between November 27, 2023, and January 5, 2024, Cloudflare took the following actions:

• Engaged Crowdstrike to do a parallel investigation to confirm Cloudflare’s findings or find anything else they missed (Crowdstrike confirmed it all).

• Conducted forensic triage of 4,893 systems.

• Rotated every production credential (>5K creds).

• Reimaged and rebooted every machine in their global network.

• Physically segmented the test and staging systems.

• Returned all data center hardware in Brazil to the manufacturer to have them confirm whether any of the devices had been modified (they found nothing).

• Reviewed updated and unused active employee accounts to confirm their legitimacy.

• Searched for secrets in JIRA tickets, source code, and HAR files. They changed anything they found.

While those actions have been completed, the company continues to explore areas to harden the environment. Kudos to Cloudflare for one hell of a detection and response effort and their transparency in all of it.

News
What Else is Happening?

🇨🇳 The FBI removed KV Botnet malware from hundreds of infected home routers. The malware was used by Chinese state-sponsored attackers to conduct attacks against US critical infrastructure.

💻️ The Remote Desktop Application provider, AnyDesk, said that their production systems were hacked, and the attackers accessed their code signing certificates. More to come on this in the coming weeks.

🛑 CISA ordered federal agencies to disconnect Ivanti VPN devices until proper investigation and remediation steps have been taken to see if threat actors (aka China, who used this against US agencies) compromised their environments.