A recent bonus issue of the Pragmatic Engineer Newsletter, authored by Gergely, delves into the impact of global configuration errors on large-scale outages. Two weeks after a significant outage, Cloudflare experienced another major disruption on December 5th, affecting thousands of sites. The root cause was a global configuration change related to a React security fix which led to HTTP 500 errors after the testing tool was disabled with a global killswitch.
Cloudflare’s response involved a swift postmortem, highlighting an impact on 28% of its HTTP traffic due to the change. The company’s efforts to harden the ingestion of configuration files have not yet been fully implemented, resulting in repeated outages. Cloudflare plans to adopt staged configuration rollouts to improve reliability, emphasizing enhanced rollouts, break glass capabilities, and fail-open error handling to mitigate the impact of such incidents.
Global configuration errors frequently lead to large-scale outages, as seen in other notable cases involving DNS changes, OS updates, and globally replicated configurations. Implementing gradual rollouts is complex but necessary for large systems to prevent these issues. However, for smaller companies, the tradeoff between speed and stability might make such changes less appealing.
This topic is one of four covered in the latest edition of The Pulse. Other articles discuss issues such as AWS’s capacity planning, the integration of Rust into the Linux kernel, and how Oxide Engineering uses LLMs. For more details, readers are encouraged to access the full newsletter.
Subscribe to the weekly Pragmatic Engineer Newsletter for insights and the latest tech updates.