Jul 7, 2019

Configuration is code

The 2019 Cloudflare outage highlighted the importance of treating configuration as code. This article explores why configuration should be managed with the same rigor as software code, discusses common pitfalls in deployment practices, and emphasizes the need for thorough testing to ensure system stability and performance.

Mar 13, 2019

Resilience lessons from aerospace disasters

One of the editions of Resilience Roundup, which is a newsletter dedicated to the topic of systems resilience, compiled by Thai Wood, talks about a paper from MIT’s Nancy Leveson called “The Role of Software in Spacecraft Accidents”. In this paper, the author studied five prominent disasters in aerospace industry with relation to software. Thai Wood does a much better job of summarizing the paper than I would, so I will not go over the whole paper here.

Mar 15, 2018

Extracting Value From Incidents With Blameless Postmortems

In any non-trivial system, failure is inevitable and incidents happen. How we deal with the incidents aftermath determines if they just cost money or we gain something from them. I’ve heard John Allspaw say a quote about incidents or outages, which fits perfectly: Incidents are unplanned investments in your company’s survival. So you have made this “investment”, everything is back up and running and now you should get as much value from it as possible.

Mar 1, 2018

Timeouts: Simple, powerful and often neglected

Timeouts are a simple yet powerful mechanism that can prevent system failures and ensure reliability. However, they are often overlooked in software development, leading to potential issues like hanging processes or resource exhaustion. This article explores the importance of setting appropriate timeouts and provides practical insights on how to implement them effectively. Learn why timeouts are crucial for building resilient and efficient systems.

Jan 22, 2018

Implementing Circuit Breaker Pattern

When you have a microservice architecture, you have to deal with failures on many different levels. One of the patterns commonly used to deal with failures of remote calls is the circuit breaker. It helps preventing cascading failures, when a problem in one services causes exhaustion of resources in other services. Let’s take a look at one example, how one might go about implementing this pattern in a Node.js microservice.

Jan 11, 2018

The four levels of resilience in systems

When we talk about complex systems, one of the studied properties is often the resilience of the system. By resilience we mean the capacity to recover from difficulties or failures. It’s often talked about in the realm of computer systems, but applies as well for others systems such as organizations. It’s helpful to classify the level of resilience, so that we can assess the current state of our system and decide if we want to invest in increasing it’s resilience, based on how critical the system is.