How CrowdStrike Took 8.5 Million Systems Offline

AI Spotlight
Deepfake Stares

Please forgive deepfake detectors for staring awkwardly into your eyes. They’re only doing it because new research coming from the University of Hull found a cool technique to detect deepfakes. It’s based on analyzing the reflection of light in your eyes.

The technique, which has roots in how astronomers study the light distribution of galaxies, is surprisingly basic.

It just looks at the reflection in each eye and sees if they match up. For a real person, they are consistent because…well, physics. Light is going to hit both eyes in a similar pattern.

Deepfakes, though, don’t live in a world constrained by physics. The researchers found that deepfakes tend to have inconsistent reflections.

So I guess this means that the eyes are the doorway to your soul and detecting deepfakes. That will make a great Valentine's Day card for your partner. No need to credit me.

Security Deep Dive
CrowdStrike Retro

 I’ll be honest. This is about to get pretty nerdy. If you’re tired of hearing about CrowdStrike or feel like you’re watching another bad infomercial, skip ahead to the end of the newsletter for a reward…

If you’re here for the nerdiness, let’s continue.

CrowdStrike CEO reported that as of July 25th, 97% of Windows systems impacted during the faulty CrowdStrike update are now operational. This is good news for many, except for Delta, which dealt with a ton of flight cancellations.

CrowdStrike released a report this week detailing what happened. On Friday, July 19, 2024, at 04:09 UTC, CrowdStrike pushed a faulty content configuration update for its Windows sensor. That update caused 8.5 million Windows systems to crash. At 05:27 UTC, CrowdStrike reverted the changes.

Any Windows system that was online, connected to the Internet, and running Crowdstrike on July 19, 2024, between 04:09 UTC and 05:27 UTC, at risk of going down.

We must understand how CrowdStrike sends updates to understand where things went wrong. Let’s start with the Windows sensor, which is the agent that sits on the Windows system and monitors for malicious activity. When you think of CrowdStrike’s Falcon Endpoint Detection and Response (EDR) agent, you think of the Falcon Windows sensor.

The Falcon Windows sensor relies on two types of security content updates to stay current on the latest threats.

The first update type is Sensor Content. These are major updates that are shipped with sensor updates. It includes AI and machine learning models that sit in the sensor and give it the brains to find badness in the system. It’s only updated when the Falcon sensor is updated, so not all too often.

Sensor Content includes a feature called Template Types, which are pre-defined fields that threat detection engineers can use to look for specific malicious behaviors on the system. It also allows for more rapid detection updates. Template Types cover categories of detections, like looking for malicious network traffic from the system.

Because Template Types are just a framework, the Falcon sensor still needs to know what type of bad behavior to look for. That’s where Rapid Response Content comes into play. Rapid Response Content allows the sensor to observe, detect, or prevent malicious activity on the system.

Rapid Response Content updates are sent to systems through Template Instances, a type of Template Type. These can occur multiple times a day.

Putting all of that together, here’s the flow: Rapid Response Content → Template Instance → Template Type → Sensor Content → Falcon Sensor

Updates through this flow are achieved with the following:

Content Configuration System: Creates Template Instances (what delivers the Rapid Response Content updates to systems). This sits in CrowdStrike’s cloud. These Template Instances are sent to Falcon sensors through Channel Files.

Content Interpreter: This is part of the Falcon sensor. It reads the Channel File and interprets the Rapid Response Code. It is also supposed to “gracefully” handle errors from bad content. More on this later.

Sensor Detection Engine: Also part of the Falcon sensor, this takes the Rapid Response Code and does the work in observing, detecting, or preventing malicious activity on the endpoint.

Cool. I get how CrowdStrike updates content. Don’t they test it? Yes, they do—quite a bit, too. Here’s a timeline to show how far back all the tests go and where things went wrong…

• February 28, 2024: Sensor Content that included a new Template Type to detect emerging attacks was released.

• March 5, 2024: The new Template Type, which covered a “variety of operating systems and workloads,” passed a stress test. The Template Type released in the February 28th update also passed.

• March 5, 2024: A new template Instance was released in the now infamous Channel File 291 (the file that everyone needed to delete if they couldn’t restart their Windows system).

• April 8 - April 24, 2024: Three additional Template Instances deployed and performed as expected.

• July 19, 2024, 04:09 UTC: CrowdStrike released two additional Template Instances. Both files passed validation due to a bug in the Content Validator, even though one of them contained faulty code, which ultimately caused Windows systems to crash.

• July 19, 2024, 05:27 UTC: CrowdStrike reverts the update.

The faulty update went undetected because of initial tests on the Template type on March 5, 2024, the previous Template Instance deployments, and a bug in the Content Validator.

When the bad Rapid Response Content was delivered to the sensor, it passed the checks, and when read by the Content Interpreter, it triggered an unexpected exception that Windows couldn’t handle and resulted in the Windows crashing with a blue screen of death (BSOD).

How is CrowdStrike fixing this moving forward? They outlined a slew of changes that included:

• Enhanced software testing procedures for Rapid Response Content and better validation checks for the Content Validator

• Enhanced resilience and recoverability to manage unexpected errors better than how you handle family at Thanksgiving.

• Refined deployment strategy to stagger deployment and avoid impacting 8.5 million systems at the same time again…

• Have third parties review their code and quality processes to find areas of improvement.

Oof, that was a long write-up. Did anyone actually make it this far? If you did, good to see you. I’m not even going to bother with a better conclusion than this 🎤 ⬇️ 

Security News
What Else is Happening?

🥶 Russian hackers targeted a municipal energy company and shut off heat to over 600 apartment buildings for nearly 48 hours. The attack involved malware designed to interact with industrial control systems (ICS) over Modbus, a communication protocol created in 1979. The malware reported incorrect measurements, which shut off customers’ heating.

🚰 A hacker leaked the database from the first BreachForums hacking site (circa 2022). It included the member ID, login name, email address, registration IP address, last IP address used while visiting the site, and private messages for all 212,414 forum members.

🤖 Cyber security awareness company KnowBe4 published details on how they accidentally hired a North Korean attacker. They were tipped off after the employee received the laptop and immediately started loading malware, which their security teams detected.

💔 Per The Record, veteran cybercriminals are “shying away” from large ransomware-as-a-service platforms due to ongoing law enforcement operations and ALPHV/BlackCat’s exit scam. Perhaps we’ll see a shift in attack numbers…or maybe it will be much harder to track moving forward.