CrowdStrike update BSOD calamity (worldwide)

[Artemis]

CrowdStrike update BSOD calamity (worldwide):

"Global IT outage live: Computer havoc caused by Crowdstrike outage could take days to fix"
https://www.abc.net.au/news/2024-07-...alia/104119960

"Global tech outages hit airlines, banks and businesses"
https://edition.cnn.com/business/liv...hnk/index.html

"Travel, banking and businesses hit after software bug causes worldwide IT chaos"
https://www.bbc.com/news/live/cnk4jdwp49et

"CrowdStrike Windows Outage—What Happened And What To Do Next"
https://www.forbes.com/sites/kateofl...at-to-do-next/

"Major Windows BSOD issue takes banks, airlines, and broadcasters offline / A faulty update from cybersecurity provider CrowdStrike is responsible for the global outage."
https://www.theverge.com/2024/7/19/2...e-outage-issue

"BSOD error in latest crowdstrike update":
https://www.reddit.com/r/crowdstrike...strike_update/

[ASAP]

it seems like there is some new tech related mess all the time now...

i am not a tech guy but Crowdstrike is a cybersecurity company... presumably if they have an outage... i don't see how this affects the core systems of a number of companies... unless it is a hack?

[RockCrusher]

Originally Posted by ASAP

it seems like there is some new tech related mess all the time now...

i am not a tech guy but Crowdstrike is a cybersecurity company... presumably if they have an outage... i don't see how this affects the core systems of a number of companies... unless it is a hack?

Word is that a software update gone wrong brought down CrowdStrike.

[RickFLM4]

Originally Posted by ASAP

it seems like there is some new tech related mess all the time now...

i am not a tech guy but Crowdstrike is a cybersecurity company... presumably if they have an outage... i don't see how this affects the core systems of a number of companies... unless it is a hack?

It's almost like concentrating software services, especially security-related services, in a few (or one) big players that serve entire industries and affect PCs everywhere, might have some downsides.

[zx10guy]

Read through the Forbes article on this. It's from a faulty update from Crowdstrike. I don't have any personal hands on experience with Crowdstrike. But from what I gather, the installed agents on the end points do automatic updates when available from Crowdstrike. Crowdstrike is a cloud based security platform. So this is why the impact has been pretty massive.

A fix has been identified but requires booting into Safe mode and then going to a specific directory on the PC/server to delete a specific file. So far there's no automated way of doing this so it's going to be a long manual process until Crowdstrike figures something out.

This is what we in the IT world call an RGE (resume generating event). And someone(s) is going to have a bad meeting with management/HR. I'm stunned that this wasn't identified in beta testing before being pushed out as a production/general release. Don't know how Crowdstrike is going to handle the ire of customers with real significant monetary loses due to this.

[Frostynorth]

On the plus side, if the computers can't boot, they can't get hacked.

[NoMoreVauxhalls]

Back in 2004 the computer security company that I worked for had exactly the same issue. (I won't name the company, but it was one of the big ones). We produced virus definition files every day (sometimes multiple times per day) which allowed the AV software to detect new viruses. These definition files are (were) created by an automated process and were QA tested before being deployed into the field - but due to the quick turnaround, corners were obviously cut during QA.

On that particular day, the update was created and started being downloaded and consumed by our software on computers around the world. Unfortunately, we incorrectly identified a Windows system file as being infected, and so that file was "quarantined" and moved from it's System folder, to our quarantine location.

End result: the sh!t hit the fan. And quickly. Any Windows PC would blue screen on the next bootup as that system file was no longer present. This sounds very similar to what has happened with Crowdstrike today...

I worked in tech support back then, and our phones were red hot for days! We subsequently revised our QA processes and made them much more robust. Individuals lost their jobs, also. And I'm guessing something similar will happen at Crowdstrike...

[Artemis]

Originally Posted by RickFLM4

It's almost like concentrating software services, especially security-related services, in a few (or one) big players that serve entire industries and affect PCs everywhere, might have some downsides.

(source: https://www.abc.net.au/news/2024-07-...alia/104119960)

[vreihen16]

As I said to my DW when I heard the talking heads on the morning TV news opening with this news as their headline, I'm glad that I was involuntarily retired from the IT business a few weeks ago.

Back to my nap.....

[unluky]

Secret Service right now going WHEW!!!!!!

[zx10guy]

Originally Posted by NoMoreVauxhalls

Back in 2004 the computer security company that I worked for had exactly the same issue. (I won't name the company, but it was one of the big ones). We produced virus definition files every day (sometimes multiple times per day) which allowed the AV software to detect new viruses. These definition files are (were) created by an automated process and were QA tested before being deployed into the field - but due to the quick turnaround, corners were obviously cut during QA.

On that particular day, the update was created and started being downloaded and consumed by our software on computers around the world. Unfortunately, we incorrectly identified a Windows system file as being infected, and so that file was "quarantined" and moved from it's System folder, to our quarantine location.

End result: the sh!t hit the fan. And quickly. Any Windows PC would blue screen on the next bootup as that system file was no longer present. This sounds very similar to what has happened with Crowdstrike today...

I worked in tech support back then, and our phones were red hot for days! We subsequently revised our QA processes and made them much more robust. Individuals lost their jobs, also. And I'm guessing something similar will happen at Crowdstrike...

While details will probably solidify as to what the issue ultimate is/was, based on the info from the Forbes article for the fix:

1. Boot Windows into Safe Mode or WRE.

2. Go to C:\Windows\System32\drivers\CrowdStrike

3. Locate and delete file matching "C-00000291*.sys"

4. Boot normally.

This isn't a Windows system file that was mistakenly determined as a malicious file. This all points to an update CrowdStrike pushed down that caused an issue with their software that has hooks into the Windows kernel that is causing the BSOD and boot loops.

[spazzyfry123]

Things aren’t so great here on the healthcare side of the fence…

[NoMoreVauxhalls]

Originally Posted by zx10guy

This isn't a Windows system file that was mistakenly determined as a malicious file. This all points to an update CrowdStrike pushed down that caused an issue with their software that has hooks into the Windows kernel that is causing the BSOD and boot loops.

You're correct - it's not exactly the same issue. But I just wanted to relay a very similar (and very painful) issue that we had back in the day!

Hopefully Crowdstrike learn a hard lesson from this and fully review their QA processes going forward.

[Artemis]

https://www.crowdstrike.com/blog/sta...sensor-update/

[JMcLellan]

Originally Posted by zx10guy

I'm stunned that this wasn't identified in beta testing before being pushed out as a production/general release. Don't know how Crowdstrike is going to handle the ire of customers with real significant monetary loses due to this.

100% this. How does this pass testing? With what is at stake and the potential impact it's crazy. It's hard to accept their QA was that bad and it's easy to go to the thought that a hack might be likely. I bet Crowdstrike wishes they could say this was a hack vs an internal issue. They may never recover.

[Artemis]

(source: https://www.telegraph.co.uk/business...stralia-world/)

Newark, USA earlier today: