Jul 7, 2019

Configuration is code

You probably experienced it yourself, or at least heard about it. The internet broke for half an hour on July 2nd, 2019. What really happened was Cloudflare, a major CDN (content delivery network) provider, had a global outage. But it effectively broke the internet, because majority of websites either use Cloudflare directly, or depend on a service which uses it. This shows us how the whole web is getting more and more centralized and if one of the major players has issues, everybody feels it.

Mar 13, 2019

Resilience lessons from aerospace disasters

One of the editions of Resilience Roundup, which is a newsletter dedicated to the topic of systems resilience, compiled by Thai Wood, talks about a paper from MIT’s Nancy Leveson called “The Role of Software in Spacecraft Accidents”. In this paper, the author studied five prominent disasters in aerospace industry with relation to software. Thai Wood does a much better job of summarizing the paper than I would, so I will not go over the whole paper here.

Mar 15, 2018

Extracting Value From Incidents With Blameless Postmortems

In any non-trivial system, failure is inevitable and incidents happen. How we deal with the incidents aftermath determines if they just cost money or we gain something from them. I’ve heard John Allspaw say a quote about incidents or outages, which fits perfectly: Incidents are unplanned investments in your company’s survival. So you have made this “investment”, everything is back up and running and now you should get as much value from it as possible.

Mar 1, 2018

Timeouts: Simple, powerful and often neglected

The story I am sure this has never happened to you, but try to imagine. Your on-call phone just got the message: The site is down! You open your Nagios dashboard and you are greeted by a sea of red. All Apache servers are not responding. The database connections are exhausted. Load balancers are screaming that they have no healthy backends. Everything else looks dead quiet. Everybody starts throwing around their guesses, what is going on.

Jan 22, 2018

Implementing Circuit Breaker Pattern

When you have a microservice architecture, you have to deal with failures on many different levels. One of the patterns commonly used to deal with failures of remote calls is the circuit breaker. It helps preventing cascading failures, when a problem in one services causes exhaustion of resources in other services. Let’s take a look at one example, how one might go about implementing this pattern in a Node.js microservice.

Jan 11, 2018

The four levels of resilience in systems

When we talk about complex systems, one of the studied properties is often the resilience of the system. By resilience we mean the capacity to recover from difficulties or failures. It’s often talked about in the realm of computer systems, but applies as well for others systems such as organizations. It’s helpful to classify the level of resilience, so that we can assess the current state of our system and decide if we want to invest in increasing it’s resilience, based on how critical the system is.