CrowdStrike is blaming a bug in an update that allowed its cybersecurity systems to push bad data out to millions of customer computers, setting off last week’s global tech outage that grounded flights, took TV broadcasts off-air and disrupted banks, hospitals and retailers.
CrowdStrike also outlined measures it would take to prevent the problem from recurring, including staggering the roll-out of updates, giving customers more control over when and where they occur, and providing more details about the updates that it plans.
The company on Wednesday posted details online from its “preliminary post incident review” of the outage, which caused chaos for the many businesses that pay for the cybersecurity firm’s software services.
The problem involved an “undetected error” in the content configuration update for its Falcon platform affecting Windows machines, the Texas company said.
A bug in the content validation system allowed “problematic content data” to be deployed to CrowdStrike’s customers. That triggered an “unexpected exception” that caused a Windows operating system crash, the company said.
As part of the new prevention measures, CrowdStrike said it’s also strengthening internal testing as well as putting in place “a new check” to stop “this type of problematic content” from being deployed again.
The US company is trying to piece together the series of events that led to crashed Microsoft Windows computer systems around the world, taking down airline, banking and stock exchange operations from Australia and Japan to the UK.
CrowdStrike has said a “significant number” of the approximately 8.5 million computers that crashed on Friday, causing global disruptions, are back in operation as customers and regulators await a more detailed explanation of what went wrong.
But for several hours, bankers in Hong Kong, doctors in the UK and emergency responders in New Hampshire found themselves locked out of programs critical to keeping their operations afloat.
Once its investigation is complete, CrowdStrike said that it will publicly release its full analysis of the meltdown.
The outage caused days of widespread technological havoc, highlighted how much of the world depends on a few key providers of computing services, and drawn the attention of regulators who want more details on what went wrong.
01:19
Computer system breakdown at Hong Kong airport delays dozens of Cathay flights, stranding travellers
Computer system breakdown at Hong Kong airport delays dozens of Cathay flights, stranding travellers
Still, the power that this mistake had to hobble critical businesses and services worldwide last week has raised fears about the vulnerability of the global IT system, which is dependent on a handful of dominant tech companies.
CrowdStrike’s shares dropped nearly 30 per cent in the aftermath of the outage, slashing billions of dollars from its market value.
The US House Committee on Homeland Security requested an appearance from Chief Executive Officer George Kurtz, and lawmakers called on him to explain how the company will mitigate risks of a similar incident in the future.
Shawn Henry, CrowdStrike’s chief security officer, apologised in a post on LinkedIn on Monday, saying that the company had “failed” its customers.
“The confidence we built in drips over the years was lost in buckets within hours, and it was a gut punch,” he said.