Crowdstrike published a faulty update. Causes Windows to bluescreen. Driver is C-00000291*.sys. Will cause worldwide outages. Thread follows, I suspect. 🧵
If anybody is wondering the impact of the Crowdstrike thing - it’s really bad. Machines don’t boot.
The recovery is boot in safe mode, log in as local admin and delete things - which isn’t automateable. Basically Crowdstrike will be in very hot water.
Favour to IT folks fixing - could you please copy the C-00000291*.sys file to somewhere and upload it to Virustotal, and reply with the Virustotal link or file hash? It's still unclear if the update was malicious or just a bug.
The .sys files causing the issue are channel update files, they cause the top level CS driver to crash as they're invalidly formatted. It's unclear how/why Crowdstrike delivered the files and I'd pause all Crowdstrikes updates temporarily until they can explain.
This is going to turn out to be the biggest 'cyber' incident ever in terms of impact, just a spoiler, as recovery is so difficult.
I'm seeing people posting scripts for automated recovery.. Scripts don't work if the machine won't boot (it causes instant BSOD) -- you still need to manually boot the system in safe mode, get through BitLocker recovery (needs per system key), then execute anything.
Crowdstrike are huge, at a global scale that's going to take.. some time.
For anybody wondering why Microsoft keep ending up in the frame, they had an Azure outage and- this may be news to some people- a lot of Microsoft support staff are actually external vendors, eg TCS, Mindtree, Accenture etc.
Some of those vendors use Crowdstrike, and so those support staff have no systems.
By far my fave thing with the Crowdstrike thing is Microsoft saying to try turning impacted PCs off and on again in a loop until you get the magic reboot where CrowdStrike updates before it blue screens.
The chuckle brothers at NoName attempting to claim they caused the incident. To be super clear, NoName can barely DDoS a bike shed website, and once asked me to make their logo in Minecraft.
The CrowdStrike outage affecting Microsoft Windows systems caused error messages worldwide on Friday. Here are some images of the weirdest ones we’ve found.
Truthfully these issues happen across all vendors - I’ve had my orgs totalled twice now by AV vendors, one while I was on holiday abroad and had to suspend said holiday.
Btw, that isn’t to excuse it or any vendor. CrowdStrike have gotta be better at this stuff. And they’ll have to, as if they aren’t transparent customers will flee.
It’s a warning shot to all AV/EDR/XDR vendors that if you fuck up availability, your brand will become failure. It’s harsh but that’s the media cycle and modern world.
Microsoft estimate almost 9 million Windows devices are impacted by the CrowdStrike incident (likely from crash telemetry). blogs.microsoft.com/blog/2024/…
The Verge has a quick look at the orgs trying to recover from the Crowdstrike incident.
If you’re wondering why it’s dropped off the radar of most press, they think it’s over as Down Detector looks okay (which, to be clear, is not good logic).
Interesting - did anybody keep a list of tweets by CrowdStrike staff during the start of the incident? This one has been deleted. x.com/brody_n77/status/1814186…
Crowdstrike are touting auto remediation of blue screen as an opt in feature.
However, I just tried it - it’s not very successful, most boots still blue screen of death. I think CS need to be careful on messaging about this as it sounds like they’re offering it as a silver bullet. It only works if networking kicks in and the agent updates before Windows finishes booting.
Microsoft's statement that a faulty CrowdStrike update affected less than 1% of active Windows systems doesn't tell the full story, since large organizations in critical sectors make up a disproportionate part of the user base, as the outages in heal…
This video for remote users with local administrator privileges, outlines the steps required to self-remediate a Windows laptop experiencing a blue screen of...
If anybody wonders what the file that took down 8.5 million Windows systems looks like.. it was 41kb in size. The only validity checking I can see CrowdStrike driver does is to check the first few bytes match the pattern seen in the screenshot before loading and executing.
The initial Post Incident Review is out from CrowdStrike. It’s good and really honest.
There’s some wordsmithing (eg channel updates aren’t code - their parameters control code).
The key take away - channel updates are currently deployed globally, instantly. They plan to change this at a later date to operate in waves. This is smart (and what Microsoft do for similar EPP updates).
By ‘this is smart’ I mean ‘this is smart… now’. Obviously they shouldn’t have been globally, simultaneously deploying kernel driver parameter changes across all customers: it was waiting to go wrong.
They still are btw, as it will take a while to engineer the correct way of doing it.
@herrman_sk Technically, it's not the driver, and AFAICT it's technically not a plugin module either, it's more like a file that a plugin module reads, and that read is a bit too trusting. @GossiTheDog
@BenAveling OK, so plugin module, the stupid question of mine still persist. The mechanics how it happened is as you've described and what I am wondering is, whether ANY change to such component, which is hooked too deep in the system, should not be run only if signed (and reviewed before) by Microsoft.
On insurance and CrowdStrike, Parametrix claim amongst just the Fortune 500 companies, they are facing $5.4bn in losses, of which around 10% will be covered by insurance. theguardian.com/technology/art…
- This year TCS migrated their EDR to CrowdStrike - Then they announced a strategic partnership with CrowdStrike - Then they lost all their systems - They’re just finishing recovery today, 6 days in - Then they got a $10 Uber Eats voucher - …which got cancelled due to Uber flagging CrowdStrike’s account as fraudulent
Questions for your EDR providers (do not assume they are experts in availability):
- What are your different update processes? - How do you test them? - Do you dogfood test them? - Do you roll them out in waves? What are the details, eg what percentages and when? - Do you monitor failures and roll back?
Microsoft has started responding with changes it wants to see in the wake of the CrowdStrike botched update. It looks like Windows kernel access is on the agenda.
Risky Business host Patrick Gray talks to SentinelOne's Chris Krebs and Alex Stamos about CrowdStrike's baffling failure and what it means for the wider secu...
Re the Delta case - the lawyer they’ve hired successfully sued Microsoft previously on behalf of the US government, and the decision was upheld on appeal too. The ruling almost lead to the breaking up of Microsoft.
The following US government backed out of the case.
Bill Gates said at the time the lawyer was “out to destroy Microsoft”.
So there’s a chance here the CrowdStrike incident may end up having implications across vendor industry around warranties etc, we’ll see.
Spirit Airlines in the US anticipates a $7.2 million hit to its third-quarter operating income due to operational disruptions caused by the CrowdStrike incident, which forced the carrier to cancel 470 flights.
Here's the Delta boss on his thoughts about the CrowdStrike incident.
They had 40k Windows Server boxes alone, all with BitLocker full disk encryption enabled, all of which wouldn't boot and weren't fixable without manually unlocking BitLocker. That had gone all in with CrowdStrike + Microsoft's most premium offerings.
He has a really good point about how tech companies have become obsessed with growth as their only metric of success, and customer satisfaction is not on the radar.
There's a really mad moment in that interview where they ask them what assistance CrowdStrike have offered, and he essentially says nothing, not even a lunch voucher.
Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •If anybody is wondering the impact of the Crowdstrike thing - it’s really bad. Machines don’t boot.
The recovery is boot in safe mode, log in as local admin and delete things - which isn’t automateable. Basically Crowdstrike will be in very hot water.
Kevin Beaumont
in reply to Kevin Beaumont • • •You know it was coming...
Crowdstrike's BSOP theme tune
Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •I've obtained copies of the .sys driver files Crowdstrike customers have. They're garbage. Each customer appears to have a different one.
They trigger an issue that causes Windows to blue screen.
I am unsure how these got pushed to customers. I think Crowdstrike might have a problem.
For any orgs in recovery mode, I'd suspend auto updates of CS for now.
Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •Chaos persists as IT outage could take time to fix, says cybersecurity firm boss
BBC NewsKevin Beaumont
in reply to Kevin Beaumont • • •The .sys files causing the issue are channel update files, they cause the top level CS driver to crash as they're invalidly formatted. It's unclear how/why Crowdstrike delivered the files and I'd pause all Crowdstrikes updates temporarily until they can explain.
This is going to turn out to be the biggest 'cyber' incident ever in terms of impact, just a spoiler, as recovery is so difficult.
Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •I'm seeing people posting scripts for automated recovery.. Scripts don't work if the machine won't boot (it causes instant BSOD) -- you still need to manually boot the system in safe mode, get through BitLocker recovery (needs per system key), then execute anything.
Crowdstrike are huge, at a global scale that's going to take.. some time.
Kevin Beaumont
in reply to Kevin Beaumont • • •Crowdstrike statement: bbc.co.uk/news/live/cnk4jdwp49…
Basically 'it's not a security incident... we just bricked a million systems'
Chaos persists as IT outage could take time to fix, says cybersecurity firm boss
BBC NewsKevin Beaumont
in reply to Kevin Beaumont • • •For anybody wondering why Microsoft keep ending up in the frame, they had an Azure outage and- this may be news to some people- a lot of Microsoft support staff are actually external vendors, eg TCS, Mindtree, Accenture etc.
Some of those vendors use Crowdstrike, and so those support staff have no systems.
But MS isn’t the outage cause today.
Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •BBC News at 6 is leading the entire show with this. (They asked me to appear but I was slightly busy).
For the record I spent much of the day trying to tell people it isn’t a Microsoft issue.
Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •Billboards in Times Square blue screen of deathing. Nice way to find out which orgs use Crowdstrike, this 🤣
Source is BBC News, if anybody wondering.
Kevin Beaumont
in reply to Kevin Beaumont • • •Chaos persists as IT outage could take time to fix, says cybersecurity firm boss
BBC NewsKevin Beaumont
in reply to Kevin Beaumont • • •CrowdStrike outage: Windows blue screen photos from around the world
William Joel (The Verge)Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •CrowdStrike have effectively a mini root cause analysis out
Pretty much as everybody knows, they did a channel update and it caused the driver to crash.
If they blame the person who did the update.. they shouldn’t, as it sounds like an engine defect.
crowdstrike.com/blog/technical…
Technical Details: Falcon Update for Windows Hosts | CrowdStrike
CrowdStrikeKevin Beaumont
in reply to Kevin Beaumont • • •For the people thinking ‘shouldn’t testing catch this?’, the answer is yes. Clearly something went wrong.
This isn’t CrowdStrike’s first rodeo on this, although it is the most severe incident so far.
Eg just last month they had an issue where a content update pushed CPU to 100% on one core: thestack.technology/crowdstrik…
Truthfully these issues happen across all vendors - I’ve had my orgs totalled twice now by AV vendors, one while I was on holiday abroad and had to suspend said holiday.
CrowdStrike bug maxes out 100% of CPU, requires Windows reboots
The StackKevin Beaumont
in reply to Kevin Beaumont • • •Btw, that isn’t to excuse it or any vendor. CrowdStrike have gotta be better at this stuff. And they’ll have to, as if they aren’t transparent customers will flee.
It’s a warning shot to all AV/EDR/XDR vendors that if you fuck up availability, your brand will become failure. It’s harsh but that’s the media cycle and modern world.
Kevin Beaumont
in reply to Kevin Beaumont • • •Helping our customers through the CrowdStrike outage - The Official Microsoft Blog
David Weston (The Official Microsoft Blog)Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •The Verge has a quick look at the orgs trying to recover from the Crowdstrike incident.
If you’re wondering why it’s dropped off the radar of most press, they think it’s over as Down Detector looks okay (which, to be clear, is not good logic).
theverge.com/2024/7/21/2420296…
CrowdStrike outage: Photos, videos, and tales of IT workers fixing BSODs
Wes Davis (The Verge)Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •House committee calls on CrowdStrike CEO to testify on global outage
Cristiano Lima-Strong (The Washington Post)Kevin Beaumont
in reply to Kevin Beaumont • • •Crowdstrike are touting auto remediation of blue screen as an opt in feature.
However, I just tried it - it’s not very successful, most boots still blue screen of death. I think CS need to be careful on messaging about this as it sounds like they’re offering it as a silver bullet. It only works if networking kicks in and the agent updates before Windows finishes booting.
reddit.com/r/sysadmin/comments…
Kevin Beaumont
in reply to Kevin Beaumont • • •CrowdStrike Disruption Restoration Is Taking Time
www.bankinfosecurity.comKevin Beaumont
in reply to Kevin Beaumont • • •CrowdStrike have published a video on YouTube about how to remediate PCs: youtube.com/watch?v=Bn5eRUaMZX…
(Despite the name, Self-Remediation, it is manual).
CrowdStrike Host Self-Remediation for Remote Users with Local Administrator Privileges
YouTubeKevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •Upguard have published a list of companies they say are impacted by the CrowdStrike 'Global IT Outage', based on public reporting.
upguard.com/crowdstrike-outage
Edit: obviously it’s missing most companies as most companies aren’t disclosing publicly.
Companies impacted by CrowdStrike outage
www.upguard.comKevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •The US Department of Transport has opened an investigation into Delta over the disruption related to CrowdStrike incident.
Good luck to the CrowdStrike account manager for Delta.
Kevin Beaumont
in reply to Kevin Beaumont • • •The initial Post Incident Review is out from CrowdStrike. It’s good and really honest.
There’s some wordsmithing (eg channel updates aren’t code - their parameters control code).
The key take away - channel updates are currently deployed globally, instantly. They plan to change this at a later date to operate in waves. This is smart (and what Microsoft do for similar EPP updates).
crowdstrike.com/falcon-content…
Falcon Content Update Remediation and Guidance Hub | CrowdStrike
CrowdStrikeKevin Beaumont
in reply to Kevin Beaumont • • •By ‘this is smart’ I mean ‘this is smart… now’. Obviously they shouldn’t have been globally, simultaneously deploying kernel driver parameter changes across all customers: it was waiting to go wrong.
They still are btw, as it will take a while to engineer the correct way of doing it.
Ľuboš Moščovič :donor: :rebelverified:
in reply to Kevin Beaumont • • •Ben Aveling
in reply to Ľuboš Moščovič :donor: :rebelverified: • • •Ľuboš Moščovič :donor: :rebelverified:
in reply to Ben Aveling • • •OK, so plugin module, the stupid question of mine still persist.
The mechanics how it happened is as you've described and what I am wondering is, whether ANY change to such component, which is hooked too deep in the system, should not be run only if signed (and reviewed before) by Microsoft.
Kevin Beaumont
in reply to Kevin Beaumont • • •theguardian.com/technology/art…
CrowdStrike global outage to cost US Fortune 500 companies $5.4bn
Nick Robins-Early (The Guardian)Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •If you want to know something crazy:
- This year TCS migrated their EDR to CrowdStrike
- Then they announced a strategic partnership with CrowdStrike
- Then they lost all their systems
- They’re just finishing recovery today, 6 days in
- Then they got a $10 Uber Eats voucher
- …which got cancelled due to Uber flagging CrowdStrike’s account as fraudulent
Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •Questions for your EDR providers (do not assume they are experts in availability):
- What are your different update processes?
- How do you test them?
- Do you dogfood test them?
- Do you roll them out in waves? What are the details, eg what percentages and when?
- Do you monitor failures and roll back?
Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •2024-07-22 CrowdStrike Holdings, Inc. Cybersecurity Incident
www.board-cybersecurity.comKevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •Microsoft are talking about changes to Windows after the CrowdStrike incident. Good.
theverge.com/2024/7/26/2420671…
Microsoft calls for Windows changes and resilience after CrowdStrike outage
Tom Warren (The Verge)Kevin Beaumont
in reply to Kevin Beaumont • • •There’s a really good discussion on @riskybusiness’s YouTube show about the CrowdStrike incident.
About the 3 minute mark @alex made me realise I was far too kind to CrowdStrike. He rightly rips them apart.
youtu.be/EGRqtscp4eE
Why CrowdStrike's Baffling BSOD Disaster Was Avoidable
YouTubeKevin Beaumont
in reply to Kevin Beaumont • • •Delta are looking to sue CrowdStrike and Microsoft. HT @hrbrmstr
cnbc.com/2024/07/29/delta-hire…
Delta hires David Boies to seek damages from CrowdStrike, Microsoft after outage
Jordan Novet (CNBC)Kevin Beaumont
in reply to Kevin Beaumont • • •Re the Delta case - the lawyer they’ve hired successfully sued Microsoft previously on behalf of the US government, and the decision was upheld on appeal too. The ruling almost lead to the breaking up of Microsoft.
The following US government backed out of the case.
Bill Gates said at the time the lawyer was “out to destroy Microsoft”.
So there’s a chance here the CrowdStrike incident may end up having implications across vendor industry around warranties etc, we’ll see.
Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •Replacing an XDR platform at scale takes some time, so if you’re wondering what the translation of Elon’s tweet about Crowdstrike is:
Elon: can we replace Crowdstrike?
Somebody: yes, we’ll begin looking into it but..
Elon: job done
Of course.. given how the Twitter takeover happened maybe he just got them to uninstall it and #yolosec
Kevin Beaumont
in reply to Kevin Beaumont • • •Delta’s CEO has confirmed they plan to take legal action against CrowdStrike after incurring a $500m loss
6 minute video interview: cnbc.com/2024/07/31/delta-ceo-…
Delta CEO says CrowdStrike-Microsoft outage cost the airline $500 million
Leslie Josephs (CNBC)Kevin Beaumont
in reply to Kevin Beaumont • • •CrowdStrike: Tech firm sued by shareholders over IT global outage
João da Silva (BBC News)Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •Here's the Delta boss on his thoughts about the CrowdStrike incident.
They had 40k Windows Server boxes alone, all with BitLocker full disk encryption enabled, all of which wouldn't boot and weren't fixable without manually unlocking BitLocker. That had gone all in with CrowdStrike + Microsoft's most premium offerings.
He has a really good point about how tech companies have become obsessed with growth as their only metric of success, and customer satisfaction is not on the radar.
Kevin Beaumont
in reply to Kevin Beaumont • • •There's a really mad moment in that interview where they ask them what assistance CrowdStrike have offered, and he essentially says nothing, not even a lunch voucher.
What a time to be alive.
Kevin Beaumont
in reply to Kevin Beaumont • • •Kevin Beaumont
in reply to Kevin Beaumont • • •CrowdStrike complained to Cloudflare about a CrowdStrike parody site… and Cloudflare took it down. Without a court order. clownstrike.lol/crowdmad/
Cloudflare recently announced they have become a strategic partner with CrowdStrike: cloudflare.com/en-gb/press-rel…
Clown Services Company - Unregistered Agent, Incompliance, Welfare, Debt Market, Analog, and Imaginary-Risk Solutions
clownstrike.lol