The Daily Populous Sunday July 21st, 2024 morning edition
Authored by nbcnews.com and submitted by NBCspec

SAN FRANCISCO — Security experts said CrowdStrike’s routine update of its widely used cybersecurity software, which caused clients’ computer systems to crash globally on Friday, apparently did not undergo adequate quality checks before it was deployed.

The latest version of its Falcon Sensor software was meant make CrowdStrike clients’ systems more secure against hacking by updating the threats it defends against. But faulty code in the update files resulted in one of the most widespread tech outages in recent years for companies using Microsoft’s Windows operating system.

Global banks, airlines, hospitals and government offices were disrupted. CrowdStrike released information to fix affected systems, but experts said getting them back online would take time as it required manually weeding out the flawed code.

“What it looks like is, potentially, the vetting or the sandboxing they do when they look at code, maybe somehow this file was not included in that or slipped through,” said Steve Cobb, chief security officer at Security Scorecard, which also had some systems impacted by the issue.

Problems came to light quickly after the update was rolled out on Friday, and users posted pictures on social media of computers with blue screens displaying error messages. These are known in the industry as “blue screens of death.”

Patrick Wardle, a security researcher who specializes in studying threats against operating systems, said his analysis identified the code responsible for the outage.

The update’s problem was “in a file that contains either configuration information or signatures,” he said. Such signatures are code that detects specific types of malicious code or malware.

“It’s very common that security products update their signatures, like once a day... because they’re continually monitoring for new malware and because they want to make sure that their customers are protected from the latest threats,” he said.

The frequency of updates “is probably the reason why (CrowdStrike) didn’t test it as much,” he said.

It’s unclear how that faulty code got into the update and why it wasn’t detected before being released to customers.

“Ideally, this would have been rolled out to a limited pool first,” said John Hammond, principal security researcher at Huntress Labs. “That is a safer approach to avoid a big mess like this.”

Other security companies have had similar episodes in the past. McAfee’s buggy antivirus update in 2010 stalled hundreds of thousands of computers.

But the global impact of this outage reflects CrowdStrike’s dominance. Over half of Fortune 500 companies and many government bodies such as the top U.S. cybersecurity agency itself, the Cybersecurity and Infrastructure Security Agency, use the company’s software.

h3Xx on July 20th, 2024 at 18:58 UTC »

even if they deployed straight to prod. they should have something like a canary deploy to only affect a small percentage of traffic. or deploy zones.

what a mess

yamirzmmdx on July 20th, 2024 at 16:43 UTC »

Oh, I didn't know I too was an expert.

HanlonsRazor_ on July 20th, 2024 at 16:39 UTC »

Likely skipped checks?! Really!?