American Airlines and Delta ground ALL flights as 911 and hospital systems go down across the US after worldwide tech outage

[DrQ]

American Airlines and Delta ground ALL flights as 911 and hospital systems go down across the US after worldwide tech outage​

• Customers travelling with the carriers said they had been left stranded in planes
• Hospitals, banks and airports worldwide have also been affected by outages

American Airlines and Delta ground ALL flights amid major tech outage

Major US carriers American Airlines, United and Delta have grounded all flights as hospitals and 911 services reported systems outages

www.dailymail.co.uk

 

[DrQ]

Mass IT outage hits airports, businesses and broadcasters around the world

Carriers including American Airlines, Delta Air Lines and United Airlines all issued ground stops Friday morning citing communications issues.

www.nbcnews.com

 

[Sandi Bo]

I heard American has resolved issues as of 5am. I hope so, I am flying to day :-(

 

[Brett]

They forgot to restart after the update

 

[Talent312]

Someone's gonna get sued.

 

[jd2601]

I am at LAX. Quite a mess. No signage, gates or many of the departure displays. Long lines for checkin. I am fortunate with Clear to miss the checkin mess. Delta app not updating so has incorrect information.

 

[x3 skier]

Reminds me of the guy who pulled the plug in the control tower on Airplane.

 

[simpsontruckdriver]

Someone's gonna get sued.

It is 100% Microsoft, they gave the corporation an update which caused it. Will they get sued? Probably not. I mean, how often has MS sent users an update, only to see the updates take down their PCs?

TS

 

[HitchHiker71]

Crowdstrike is at the root of this worldwide outage - their stock is down over 20% in pre-market trading:

https://www.cnbc.com/2024/07/19/what-is-crowdstrike-crwd-and-how-did-it-cause-global-it-outages.html

 

[HitchHiker71]

It is 100% Microsoft, they gave the corporation an update which caused it. Will they get sued? Probably not. I mean, how often has MS sent users an update, only to see the updates take down their PCs?

TS

It was NOT 100% Microsoft - it was 100% Crowdstrike that impacted Microsoft downstream. Microsoft has since implemented a zero day fix for impacted Crowdstrike users. Only Crowdstrike customers were impacted - however since Crowdstrike is a widely used cybersecurity service by many larger organizations, this issue is widely impacting those customers.

 

[DrQ]

Microsoft outage: Australian Michael Sentonas leads Crowdstrike - the company just sparked tech chaos that grounded flights, shut supermarkets and knocked banks offline​

• Massive global computer outage triggered on Friday
• Crowdstrike software clashed with a Windows update

Meet the mysterious Australian at the heart of a worldwide tech crisis

Melbourne man Michael Sentonas leads CrowdStrike, the publicly-listed cyber-security firm that caused the computer meltdown felt across the globe.

www.dailymail.co.uk

Mr Sentonas gave an interview earlier this year in which he boasted about about helping other companies deal with IT breaches and how they shouldn't attempt to 'trivialise' any problems.

'If an organisation has been breached, I'll often work with the team to coach them on how to deal with it,' he told The Age.​

​

'That could be how to deal with press, or avoiding coming out to say, 'there's a sophisticated adversary'. I try to coach people on being open and transparent about what happened and how you're dealing with it, which is so critically important to the customer,' he said.​

​

'You have to go to your customers and be upfront, and if you try to trivialise it, it won't go well.'​

Crowdstrike CEO issues apology for 'mother of all outages'

Crowdstrike has issued an apology after it caused worldwide chaos Friday after it deployed a faulty update to its software that caused a Microsoft outage.

www.dailymail.co.uk

 

[DrQ]

I was involved with patching. We always moved updates through DEV -> TEST -> STAGE -> PROD lifecycles.

Even with CI/CD and Cloud environments, we had DEV -> STAGE -> PROD environments.

We only had one O/S related conflict and it was caught long before PROD.

 

[SteveinHNL]

Trading in my Schwab account is also dead due to CRWD

 

[CO skier]

Trading in my Schwab account is also dead due to CRWD

I successfully executed two trades this morning at about 10:30 EDT in my Schwab account. My accounts were transferred over from TD Ameritrade; maybe that is a possible explanation for the difference?

 

[SteveinHNL]

I successfully executed two trades this morning at about 10:30 EDT in my Schwab account. My accounts were transferred over from TD Ameritrade; maybe that is a possible explanation for the difference?

I tried about 11:30 EDT. No workie. I contacted customer support and they confirmed trades are down but they offered to "try" to place trades for me over the phone. I declined because nothing I am doing can't wait for later.

 

[WaikikiFirst]

semi-strangely, only trouble I had is trying to voice call merrill-lynch. they must be using voip thru CRWD? zero problems on 3 different broker websites
I bought a few more sh of MSFT on the opening, down @ $433. I had the GTC order in b/c that is a pseudo-gap level. $430 seems like a level to buy a few more.

 

[WaikikiFirst]

how often has MS sent users an update, only to see the updates take down their PCs?

I cringe every time my PC tells me it is time to install the new update they just downloaded. and they almost always download on Tuesdays. Why Tuesday?
I wait to Sat am, sometimes a week or 2 later.

 

[DrQ]

It is 100% Microsoft, they gave the corporation an update which caused it. Will they get sued? Probably not. I mean, how often has MS sent users an update, only to see the updates take down their PCs?

TS

I cringe every time my PC tells me it is time to install the new update they just downloaded. and they almost always download on Tuesdays. Why Tuesday?
I wait to Sat am, sometimes a week or 2 later.

Enterprise versions of Windows have control of when and what updates are deployed.

 

[jp10558]

I cringe every time my PC tells me it is time to install the new update they just downloaded. and they almost always download on Tuesdays. Why Tuesday?
I wait to Sat am, sometimes a week or 2 later.

Tuesdays are the patch day. Working in IT, my guesstimate (and why we have Tuesday downs / updates) is -

No one wants to do anything risky on a Friday (Hah!) because who wants to come in on the weekend to fix a botch change?

No one wants to do anything on a Monday because you are getting back up to speed from the weekend and cleaning up any non emergencies that happened then.

Given those 2 things, you want to do changes early in the week so you have the most workdays to fix issues possible before hitting another weekend. So ... Tuesday.

 

[jp10558]

I was involved with patching. We always moved updates through DEV -> TEST -> STAGE -> PROD lifecycles.

Even with CI/CD and Cloud environments, we had DEV -> STAGE -> PROD environments.

We only had one O/S related conflict and it was caught long before PROD.

I can't imagine there was any sort of testing / QA done on these crowdstrike updates for it to go through like this. Or "conspiracy theory hat time" - malicious dev pushed through and didn't follow protocol for some reason. Even then, it really ought not be possible to have one dev go around a testing protocol either. So - either way, this comes down to bad processes IMO.

 

[DrQ]

I can't imagine there was any sort of testing / QA done on these crowdstrike updates for it to go through like this. Or "conspiracy theory hat time" - malicious dev pushed through and didn't follow protocol for some reason. Even then, it really ought not be possible to have one dev go around a testing protocol either. So - either way, this comes down to bad processes IMO.

There is a big push to implement CI/CD (Continuous Improvement / Continuous Delivery) and I've seen arguments to compress environments to speed deployments. I suspect this may be a factor.

 

[jp10558]

There is a big push to implement CI/CD (Continuous Improvement / Continuous Delivery) and I've seen arguments to compress environments to speed deployments. I suspect this may be a factor.

I know, I'm too oldskool to think move fast and break things and fix it later is really a great way to do stuff.

 

[Superchief]

Here is an explanation from a tech savie person on another forum that I follow.

"So the TLDR version of what happened, as I understand the problem - There's a security/antivirus/managed service provider company called CloudStrike. Large corporations contract with them to keep their computers up to date and virus free. This is a very common practice and is much better than the alternative of not having things be updated properly.

This company has something called a kernel level driver that interacts with the base level of the operating system to keep tabs on everything and make sure that it's working properly, as well as have access to know if programs are being naughty. They pushed an update that had a glitch in it and because it's kernel level, this causes a 'blue screen of death' The only way to fix this is to go around physically to every computer and push the F8 button when the computer is booting up and open up a command prompt and type something in to fix it.

This has to be done physically to every single computer with the problem - including servers in data centers, everything everywhere that has a problem. Somebody has to actually go to that computer to fix it, It can't be done remotely. In a data center, there may be a management console but it still has to be done for each server. Additionally, if they're using BitLocker drive encryption, a long and complicated key must be typed in. That is different for every computer that is using BitLocker. The IT guy must have this key, as well as type the whole thing, physically at the computer before they can fix it."

I would hate to work in IT today in a company that is effected. It appears they will have to manually reboot and reset every computer and server. MVC probably won't be effected because they are likely using older operating systems. It's scary that most major airlines, banks, government services, and hospitals can be shut down due to one bug in an 'update'. CloudStrike brags about having most Fortune 500 companies as customers, but I expect many of them to start looking for alternatives.

 

[WaikikiFirst]

CloudStrike

what do you get when Cloudflare & Crowdstrike get a room?
and they keep that encryption key in a briefcase locked to the wrist of some guy who is out getting french fries.
Next they will put a "Captcha" on the F8 key, just cuz, that is the way IT people roll

 

[HitchHiker71]

I cringe every time my PC tells me it is time to install the new update they just downloaded. and they almost always download on Tuesdays. Why Tuesday?
I wait to Sat am, sometimes a week or 2 later.

Patch Tuesday is a well known fact within the IT industry - almost all software companies follow this weekly patch schedule.


Sent from my iPhone using Tapatalk