Cascading failures can quickly bring down entire distributed systems and can be very hard to recover from, potentially leading to extended outages or downtime. Although recovering from such failures can be challenging, there are some steps we can take to reduce the risk of cascading failures.
In this article, I will discuss the following:
- What is cascading failure?
- Why is it hard to recover from?
- What are some techniques that can help protect against cascading failures?
What Is Cascading Failure?
A cascading failure is one that is exacerbated as a result of a positive feedback loop. A positive feedback loop occurs when change in one direction causes further change in the same direction. When a system is overloaded and it starts to return mostly errors, other components may respond in a manner that makes the problem worse.
Large-scale distributed systems are the most susceptible to cascading failures. As one node gets overloaded and starts failing, the remaining nodes see an increase in their load, causing them to fail as well, and so on.
For example, let’s assume we have a distributed database server with several downstream clients reading from it. The database sometimes fails, so clients have built-in retry logic. If there is a sudden burst in requests from the clients to the database and a large number of database hosts get overloaded, several clients will start seeing errors. As these clients retry the failed requests, the database servers will get even more overloaded, worsening the situation and leading to a cascading failure. Such failure is hard to recover from once it has started.
#distributed-systems #software-engineering #programming #reliability #devops