High Availability and Disaster Recovery

Introduction

Everything fails eventually: disks, servers, networks, data centres, software updates, people. High availability (HA) keeps the service running through small failures. Disaster recovery (DR) gets you back after a big one. Planning for both is what separates hobby setups from professional ones.

Availability targets

Availability is often stated as a percentage of time the service works. Higher targets allow far less downtime:

Each extra "nine" costs much more. Choose a target based on what downtime really costs the business, not on what sounds impressive.

Regions and zones

• A region is a geographic area with several availability zones, which are isolated data centres with their own power and networking.
• Spreading across zones protects against a data centre failure, with little extra complexity.
• Spreading across regions protects against a whole region failing, and can put your service nearer to users, but is more complex and costly. Data must be replicated across distance, and laws may limit where data may live.

Two key numbers

• RTO (recovery time objective): how long can you be down? The target time to restore service.
• RPO (recovery point objective): how much data can you afford to lose, measured in time? An RPO of 15 minutes means you must be able to recover to within 15 minutes of the failure.

Smaller numbers need more investment. Agree them with the business for each system, since not everything needs the same.

Recovery strategies

From cheapest and slowest to most expensive and fastest:

1. Backup and restore: keep backups, and rebuild from them (with infrastructure as code) after a disaster. Slow, with hours of recovery, but cheap.
2. Pilot light: a minimal copy of the core (such as the replicated database) is always running, and the rest is switched on when needed.
3. Warm standby: a smaller but running copy of the whole system, scaled up on failover.
4. Active-active (multi-site): full copies in several regions all serving users. Fastest recovery, and the most complex and costly.

Backups done right

• Follow the 3-2-1 rule: three copies, two kinds of media, one off site (another region or account).
• Keep backups separate and protected, for example with immutability, so ransomware or a mistake cannot delete them.
• Encrypt them.
• Test restores. A backup you have never restored is a hope, not a plan.

Design for failure

• Eliminate single points of failure. Ask of every component: what if this dies?
• Health checks and automatic failover for databases and load balancers.
• Graceful degradation: when a dependency fails, keep the core working, for example by hiding recommendations but still allowing checkout.
• Timeouts, retries and circuit breakers so a slow part does not drag everything down.
• Safe deployments: gradual rollouts and quick rollback, since changes cause many outages.

Practise

• Write runbooks for failure scenarios.
• Run game days, where you practise a failover or a restore.
• Use chaos engineering carefully: deliberately break things in a controlled way to learn how the system reacts.
• After incidents, hold blameless reviews and improve.

Try it yourself

Pick a system and decide its RTO and RPO. Choose a recovery strategy, estimate roughly what it costs relative to the others, and write down how you would test it.

Summary

• Availability targets translate into allowed downtime, and each extra nine costs far more.
• RTO and RPO define how fast and how current recovery must be, and they guide the strategy.
• Remove single points of failure, protect and test backups, and practise recovery.