The 2019 Cloudflare outage highlighted the importance of treating configuration as code. This article explores why configuration should be managed with the same rigor as software code, discusses common pitfalls in deployment practices, and emphasizes the need for thorough testing to ensure system stability and performance.
One of the editions of Resilience Roundup, which is a newsletter dedicated to the topic of systems resilience, compiled by Thai Wood, talks about a paper from MIT’s Nancy Leveson called “The Role of Software in Spacecraft Accidents”. In this paper, the author studied five prominent disasters in aerospace industry with relation to software. Thai Wood does a much better job of summarizing the paper than I would, so I will not go over the whole paper here. Go read the newsletter linked above. I’d just like to focus on three of the themes written about in the paper.
In any non-trivial system, failure is inevitable and incidents happen. How we deal with the incidents aftermath determines if they just cost money or we gain something from them.
Timeouts are a simple yet powerful mechanism that can prevent system failures and ensure reliability. However, they are often overlooked in software development, leading to potential issues like hanging processes or resource exhaustion. This article explores the importance of setting appropriate timeouts and provides practical insights on how to implement them effectively. Learn why timeouts are crucial for building resilient and efficient systems.
When you have a microservice architecture, you have to deal with failures on many different levels. One of the patterns commonly used to deal with failures of remote calls is the circuit breaker. It helps preventing cascading failures, when a problem in one services causes exhaustion of resources in other services. Let’s take a look at one example, how one might go about implementing this pattern in a Node.js microservice.
When we talk about complex systems, one of the studied properties is often the resilience of the system. By resilience we mean the capacity to recover from difficulties or failures. It’s often talked about in the realm of computer systems, but applies as well for others systems such as organizations. It’s helpful to classify the level of resilience, so that we can assess the current state of our system and decide if we want to invest in increasing it’s resilience, based on how critical the system is.
