High availability and disaster recovery

01

Failures will happen: availability numbers and single points of failure

Availability is the share of time your service was usable. 99.9% sounds pretty high, but over a year it means about 8 hours 45 minutes of downtime. Every extra 9 cuts the downtime you're allowed to a tenth, and the cost and the design get harder by about as much.

Chain things together and it goes down; put them side by side and it goes up

A service is made of several components linked together. If a response has to pass through a web server, then an app server, then a DB, all three have to be up. So the overall availability is the product of each component's availability. Chain three 99.9% components and you drop to about 99.7%. Put two identical servers side by side instead, and that tier only goes down if both do, so it climbs to 99.9999% (assuming the two fail independently). In the scene below, try doubling up each tier.

A spot where one thing stopping stops everything is called a single point of failure. Designing for high availability is the work of finding single points of failure and removing them one by one. And it's not just servers. A DB, a load balancer, a network path, even "the one person who knows how this server is set up" can be a single point of failure.

An SLA is a refund promise, not a guarantee

Each cloud service promises a monthly uptime through its SLA (service level agreement). Here's the part that trips people up: an SLA isn't a guarantee that "it will definitely run this much." It's a promise that "if we miss it, we'll give part of your bill back as credits." It doesn't cover what you lose while you're down, so your own service's availability has to come from your design. Even for the same service, the SLA is higher when you spread it across several zones than when you run a single server.

To get rid of a single point of failure, you went from one server to two. But what if those two sit in the same building?

02

How big a failure gets: servers, zones and regions

Failures come in different sizes. Sometimes one server's disk dies. Sometimes an entire data center goes down from a power outage, fire or network failure. And once in a while, a whole geographic area goes down from a natural disaster or a massive outage. The area a single failure affects is called its blast radius.

The cloud shows you this range as availability zones and regions. A region is a geographic area like Seoul or Tokyo, and inside it are several availability zones: groups of data centers, each with its own independent power, cooling and networking. Zones are close enough for synchronous replication (a few milliseconds of latency), yet far enough apart that a disaster in one won't spread to another.

Most production services use multiple zones as the baseline. They're in the same region, so latency is low enough to replicate the DB synchronously, and you survive a zone failure. You only expand to multiple regions when you also have to survive a region failure (laws and regulations, or a very large service), and how you do that is the disaster recovery strategies in chapter 7.

To decide how much you need to survive, you first need a target. How long can you afford to be down, and how much data can you afford to lose?

03

What to protect, and how much: RTO and RPO

When you design disaster recovery, there are two numbers you decide first. RPO (recovery point objective) is "how much data can you afford to lose when something fails," written as an amount of time. RTO (recovery time objective) is "how long can you afford to be down."

It's easier if you put the moment of failure in the middle. Looking back, everything since the last point where you protected your data (the last backup or replication) is the data you lose: that's the RPO. Looking forward, everything until the service opens again is your downtime: that's the RTO. In the scene below, try changing how you protect data and how you recover.

These two targets are set by the business, not the technology. An order system loses a lot if it's down for even a minute and can't lose a single order, while an internal reporting system might be fine being down for a day. The tighter the target, the higher the cost, so you weigh losses against cost for each system. If recovering costs more than what the outage would lose you, you've set the target too high.

With the targets set, it's time to keep things from stopping. First up: how to keep servers standing when requests suddenly pour in.