Diving into the embarrassing engineering behind CrowdStrike

What caused the CrowdStrike meltdown?

CrowdStrike's meltdown was rooted in a combination of flawed engineering processes and inadequate safeguards. The key factors included a problematic content update and issues with their boot start driver.

Root Causes:

• The deployment of a content update full of zeros that bypassed validation.
• The boot start driver was designed to load this content update but failed to handle exceptions gracefully, causing system crashes.
• The lack of staggered releases, meaning updates were pushed to all users simultaneously.

Points of Failure:

1. Validation Errors: The content update wasn't validated properly.
2. No Staggered Deployment: Updates were deployed globally without phased rollouts.
3. Insufficient Testing: The update wasn't tested on enough machines before the global rollout.
4. Boot Start Driver: Released updates were loaded during boot, causing immediate issues if faulty.
5. No Input Validation: The driver didn't validate external input, leading to crashes.
6. Circumventing WHQL Certification: Skipping crucial security checks and certifications.
7. Inadequate Monitoring: Lack of systems in place to catch issues early.
8. Flawed Deployment Procedure: Updates could be pushed without proper review.
9. Poor Exception Handling: The driver crashed instead of handling problematic content gracefully.
10. Insecure Content Update Process: Updates could be compromised due to lack of digital signature requirements.
11. Delayed Rollback Mechanism: The ability to revert failed updates quickly wasn't in place.
12. No User Control Over Updates: Lack of options for users to control updates led to widespread system impacts.
13. Insufficient Communication: The subsequent communication with users and businesses was inadequate and dismissive.

Each of these failures contributed to a systemic breakdown, ultimately leading to a global disruption of services and severe impacts on multiple critical systems.

How did CrowdStrike's content update process fail?

CrowdStrike's content update process was riddled with failures, both in design and execution. The most glaring issues revolved around the flawed content update, insufficient testing, and bypassing crucial validations.

The Process and Its Failures:

1. Creating the Update:

   • The problematic update contained all zeros, indicating severe errors.
   • There was a lack of proper validation or a continuous integration (CI) system to catch such errors early.

2. Skipping WHQL Certification:

   • CrowdStrike opted to bypass Windows Hardware Quality Labs (WHQL) certification.
   • This decision allowed potentially flawed updates to be pushed without rigorous scrutiny.

3. Deploying the Update:

   • The update was rolled out globally without a staggered deployment strategy.
   • This led to millions of systems receiving the faulty update simultaneously.

4. Boot Start Driver Issues:

   • The boot start driver, which loads critical updates during the OS boot process, crashed due to invalid content.
   • The driver failed to validate the external input, leading to systemic crashes.

5. Lack of Proper Monitoring:

   • There were no effective systems to monitor or catch these issues early.
   • This prolonged the time taken to identify and rectify the problems.

CrowdStrike's defective update process, combined with poor planning and inadequate safeguards, led to a catastrophic failure impacting countless systems and businesses.

What were the consequences of the CrowdStrike failure?

The faulty content update pushed by CrowdStrike led to widespread chaos with far-reaching implications. Here are the major impacts:

• Systems Affected: Over 5,370 flights were canceled due to affected systems. This outpaces the combined cancellations for 2018 and 2019.
• Business Disruption: Hundreds of companies, including 300 of the Fortune 500, experienced system crashes and operational halts.
• Cybersecurity Implications: The failure exposed a significant attack vector, undermining trust in antivirus solutions.
• Critical Systems Impacted: 911 services and hospital equipment like MRIs faced outages, potentially risking lives.
• Financial Losses: The global disruption led to billions of dollars in losses, missed flights, and delayed operations.

This incident underlined the importance of rigorous testing, validation, and staggered releases in software deployment. It serves as a cautionary tale for the cybersecurity industry.

How did CrowdStrike respond to the incident?

CrowdStrike's response to the incident has been heavily criticized. Initially, their handling seemed inadequate given the gravity of the situation. They offered $10 Uber Eats gift cards as an apology to those affected, which many found insulting due to the extent of the disruption and losses incurred.

Public statements from CrowdStrike aimed to clarify the problem and their actions. They explained:

"On Friday, July 19th, as part of regular operations, CrowdStrike released a content configuration update for the Windows sensor to gather telemetry on possible novel threat techniques. These updates are a regular part of the dynamic protection mechanisms of the Falcon platform."

However, throughout the entire statement, they avoided directly apologizing, focusing instead on justifying their procedures. The company insisted their content interpreters were "designed to gracefully handle exceptions for potentially problematic content."

Despite the widespread criticism and tangible impacts of the incident, CrowdStrike’s official communication did little to reassure its customers, primarily offering technical explanations without taking full accountability.

What measures is CrowdStrike taking to prevent future issues?

CrowdStrike claims to be implementing several steps to avoid repeated failures, aimed primarily at improving their update processes and monitoring systems.

Proposed Measures:

• Staggered Deployments: Acknowledge their previous error, they now plan to implement phased rollouts. This method will allow updates to reach a small group of users first to ensure stability before a wider release.
• Enhanced Testing: They aim to improve rapid response content testing by integrating various methods, such as:
  • Local developer testing
  • Stress testing
  • Fuzzing and fault injection
  • Stability and content interface testing
• Better Monitoring: New measures include heightened observation for sensor and system performance, collecting real-time feedback during deployments.
• Increased User Control: Users will gain greater control over when and where rapid response content updates occur, along with access to detailed release notes.

Analysis:

While these measures appear comprehensive, their effectiveness remains to be seen. The success of these steps will largely depend on their strict implementation and ongoing commitment to validation and monitoring. Staggered deployments and extensive testing are essential, but the real test will be their execution in live environments.

What lessons can other companies learn from CrowdStrike's mistakes?

The CrowdStrike engineering failures offer several key lessons for other companies, especially those involved in software deployment and cybersecurity.

Key Takeaways:

• Rigorous Validation Processes:

  • Ensure all updates go through exhaustive validation.
  • Use continuous integration (CI) systems to catch errors early.

• Staggered Releases:

  • Implement rolling releases to a small subset first.
  • Monitor initial deployment for any issues before a wide rollout.

• Extensive Testing:

  • Conduct versatile testing like stress tests, fuzzing, and stability checks.
  • Implement both local developer and high-scale testing environments.

• Improved Monitoring:

  • Establish real-time systems to catch and address issues early.
  • Use comprehensive tools to monitor system performance continuously.

• Enhanced User Communication:

  • Maintain transparent and timely communication with users.
  • Provide detailed release notes and updates.

• WHQL Certification:

  • Avoid shortcuts and ensure crucial security certifications are not bypassed.
  • Validate all updates through certified processes to maintain system integrity.

These lessons underscore the importance of thorough testing and staggered releases. Companies can better secure their systems and maintain user trust by adhering to these best practices. Additionally, focusing on core strengths and quality can prevent feature creep and unnecessary complexity, as highlighted by Dropbox and Webflow's practices.