News of the huge CrowdStrike-Microsoft outage over the weekend is, in the parlance of the print heydays, not so much wrapping today’s fish ’n chips takeaways as it is already festering at the bottom of the bin. Such is the pace of news these days.
In fact, I bet CrowdStrike’s PR team popped a bottle of bubbly on Sunday when US President Joe Biden chose to bow out of the US elections race. But before we move on to the next newsbite, we should consider the lessons this one holds for us.
Though the problem was relatively quickly resolved, the aftershocks are still rippling through international markets and operational centres. It was, we now understand, the
single-biggest cyber incident in history.
Not an attack, mind you, but an “incident”, birthed by a defective update issued by cybersecurity provider CrowdStrike, which caused 8.5-million Windows OS machines around the world to display the dreaded “blue screen of death” (BSOD) before entering into a cycle of BSODs and restarts.
Stuck as they were in a “reboot death spiral”, as Wired calls it, many of the affected systems needed to be manually (in-person) rebooted into safe mode before the required fix could be downloaded and tech support could pause to dab their sweaty brows.
It wasn’t just a stressful Friday night/Saturday morning for your average “IT guy”, because these machines are the computing backbone to many a critical infrastructure, from finance to aviation, rail, broadcast, maritime, public service, medical and even emergency response.
One estimate (from finance publication The Motley Fool) suggests at least 62 of the Fortune 100 were CrowdStrike customers (Q1 2024), so it makes sense then that entire banks and hospitals went down, thousands of flights were cancelled, and whole fleets grounded. Some, including Delta Air Lines, were reportedly still in repair mode as the new week dawned.
Though fewer than 1% of Windows machines (temporarily) blinked out, this has been described as an actual systemic-level crash. In reporting this week, Tech Target wrote: “In many respects the outage was a real manifestation of fears that computing users had at the end of the last century with the Y2K bug. With Y2K the fear was that a bug in software systems would trigger widespread technology failures.”
That never happened, but this incident proves the point. No wonder CrowdStrike’s share price was down about 13% on Monday — a hard blow for a firm that was clocking huge trading gains the last year.
For context, 2017’s WannaCry ransomware worm attack affected about 300,000 computers. When Meta had a six-hour collapse in 2021, billions of users were cut off from their personal and business social channels, but to be clear no-one is running a hospital or airline on Facebook Business. In an auto-update world where trusted status is preordained, Covid-19 and mega TikTok trends have nothing on the “virality” of a faulty patch.
What is just so fascinating — now that the primary dangers are largely handled and we can be philosophical about it — is that what took us down here was a tiny file issued by the “good guys”, rather than an elaborate hack or similar. And, to be clear, CrowdStrike really are usually the good guys. It’s a respectable listed firm in the S&P 500, with one of the best Endpoint Detection and Response (EDR) software offerings in the world. This wasn’t some nefarious network of cyber pirates, but the team who (99.9% of the time) protect us from said ne’er-do-wells. Put that in your risk register and smoke it.
Their trusted reputation and technical excellence are precisely why they’re so widely used by firms and governments around the world ... well that, and market concentration. That is one of the uncomfortable truths this incident truly highlights: the current state of the tech ecosystem is one of amazing global interdependence, so a single fault introduced by an individual or team, just one change implemented without due diligence, can cascade through connected systems almost instantly.
Those much-feared bad actors will have been paying attention here too. As a field, hacking is notoriously opportunistic, happy to make the most of shortcuts forged by anyone, no matter what shade of hat they don.
As an aside, individuals should also be alert to the possibility that cybercriminals may try use this news in their social engineering efforts. Be wary of any emails or phone calls from people claiming to be from CrowdStrike or offering CrowdStrike solves. We’ve already seen a rush of people trying to register site addresses that closely resemble CrowdStrike’s own.
Back on topic, CrowdStrike is at the sharp end of the criticism, and rightly so. But Microsoft does have important questions to answer about the checks and controls it should have in place before a third party (or itself) can issue an update. Windows is the world’s dominant desktop operating system, with a market share of over 70%.
In this case, Linux and MacOs systems were unaffected. But CrowdStrike does have offerings for both these and other bespoke systems, so — silver linings, I guess — it could have been worse. If it had rolled out to these other software families, the impact may have been upgraded from disastrous to catastrophic. There by the grace of Sod, go we.
China was also largely unaffected by the outage, because of its focus on local procurement. The BBC reports that CrowdStrike is “hardly used there” as “very few organisations will buy software from an American firm which, in the past, has been vocal about the cybersecurity threat posed by Beijing”.
Critics of the consolidation and concentration of providers in computing will be dining out on this story for years to come, and it should — at least — prompt some serious consideration of how we procure, structure and ring-fence our systems for redundancy and single points of failure.
These are known risks, to be clear, but have perhaps been relegated to the less urgent quadrant when all the headlines focus on hacks and malware threats.
• Thompson Davy, a freelance journalist, is an impactAFRICA fellow and WanaData member.
Would you like to comment on this article?
Sign up (it's quick and free) or sign in now.
Please read our Comment Policy before commenting.