Disaster Recovery
In one line: most design chapters stop at replication. This one asks the harder question — what happens when a whole facility is gone — and is honest that the answer involves both data loss and downtime.
The architecture
This approach backs up data, application servers, and configurations to Amazon S3 in the primary zone. Zonal replication then copies this data to a secondary zone. In the event of a disaster, services are restored using the backups in the secondary zone.
Backing up application servers and configuration, not just data
The list is "data, application servers, and configurations" — and the second and third items are the ones teams forget.
Restoring a database into an empty region gives you data nobody can serve. You also need the application images, the configuration — connection strings, feature flags, ZooKeeper state, shard mappings from Lesson 8 — and the infrastructure definitions to stand it all up.
Lesson 8's partition metadata is a good example: without the table-to-partition and primary-to-replica mappings, restored MySQL instances are a pile of data with no way to route queries.
A recovery plan that only backs up data recovers a database, not a service. That distinction is exactly what the next section is about.
The transitions are the plan. A backup only tells you the data survived — it says nothing about how long you are down while restoring, or whether traffic is served meanwhile. Those are different questions, and only the state machine answers them.
Backups are not a recovery plan
Note: a DR plan requires more than just backups; it must also ensure timely data restoration.
Key questions:
| Question | Why it matters |
|---|---|
| Which data and systems are critical for recovery? | Not everything must come back at once — you need a priority order |
| How fast can we restore from the backup facility? | This is your recovery time objective, and it is measured, not assumed |
| Can all systems be fully recovered from backups? | An untested backup is a hypothesis, not a plan |
| How do we handle data created between the last backup and the disaster? | This is your recovery point objective — the data you accept losing |
The last two questions are the ones that make DR plans fail in practice
The first two are planning questions. The last two are where most plans turn out to be fiction.
"Can all systems be fully recovered from backups?" — this can only be answered by actually restoring. A backup that has never been restored is a file you hope is useful. Backups fail silently: a misconfigured job excludes a table, an encryption key is itself only stored in the region that burned down, a format changes and nobody notices. The only test is a real restore drill.
"How do we handle data created between the last backup and the disaster?" — with daily backups, the honest answer is "we lose up to 24 hours." For Quora that means questions, answers, and votes from the last day, and Lesson 11 established those are the critical, strongly-consistent data. It is a stark gap.
This is the standard RPO/RTO framing: RPO is how much data you accept losing, RTO is how long you accept being down. Daily backups plus multi-hour restore means an RPO of a day and an RTO of hours, which is a real position — just one that should be chosen rather than discovered during an incident.
The drawbacks
- Data loss: daily backups leave a gap where data created since the last backup is lost. Synchronous cross-region replication can mitigate this.
- Downtime: restoration can take hours, during which the system cannot serve queries.
Synchronous cross-region replication would fix the data loss and cost latency on every write
The proposed mitigation deserves scrutiny, because it conflicts with a decision made earlier.
Lesson 11 chose synchronous replication within a data center for critical data, justified because intra-DC latency is a millisecond or two and the user is waiting on a human anyway.
Cross-region synchronous replication is a different proposition: tens to hundreds of milliseconds, paid on every critical write. The Distributed Cache chapter's PACELC framing applies — even with no partition, you are choosing latency over consistency, permanently.
Is it affordable here? Possibly, given Lesson 1's unusually permissive budget. Adding 100 ms to a question submission that the user expects to take a moment, and whose answer arrives in hours, is arguably still invisible.
But it is a much bigger claim than the intra-DC version, and the honest position is that it trades a day of potential data loss for latency on every write — with the answer depending on how much a day of questions and answers is worth.
Different stores recover in parallel — so total time is the maximum, not the sum
"How can using different data stores for different types of data be beneficial in disaster recovery?"
When we use different data stores, we can recover various forms of data simultaneously instead of one large blob of data. As we recover data stores with variable recovery times, the final recovery time equals the maximum recovery time among all data stores.
However, using many databases also means administrators need to know the details of checkpointing or restoring them.
This is the strongest argument for heterogeneous storage in the whole chapter, and it complements Lesson 11's fault-isolation point.
Recovery parallelizes. MySQL, MyRocks, blob storage, and Kafka restore simultaneously, so RTO is the slowest one rather than their sum. With one monolithic store, everything is serial.
You can also prioritize: restore MySQL first so questions and answers are readable, and let the ML feature store catch up afterwards — the site works degraded rather than not at all. That answers the first DR question directly.
The cost is stated honestly: more stores means more restore procedures to know and to keep tested, and an untested restore procedure is the failure mode above.
Note: Amazon S3 offers high reliability, with 99.999999999% durability and 99.9% availability.
Eleven nines of durability against three nines of availability — very different guarantees
Those two numbers are worth reading together, because they are enormously far apart and mean different things.
Durability 99.999999999% — the probability the bytes still exist. Effectively, S3 does not lose data.
Availability 99.9% — the probability you can reach it right now. That is roughly 43 minutes of unavailability per month.
So during a disaster recovery, your backups are almost certainly intact but might briefly be unreachable — and the moment you need them is exactly when you cannot tolerate that.
The general point, and it applies to any storage guarantee: durability and availability are separate promises. Systems routinely offer extraordinary durability and ordinary availability, and a recovery plan has to be built around the weaker of the two.
Key takeaway
Back up data, application servers, and configuration — restoring data alone recovers a database, not a service. Backups are not a plan: the plan is answering RPO (a day of loss with daily backups) and RTO (hours of downtime), and the only real test of a backup is a restore drill. Heterogeneous stores recover in parallel, so RTO is the maximum rather than the sum, and can be prioritized to bring the site back degraded first.
Interview signal by level
| Level | What a strong answer sounds like |
|---|---|
| L4 | "Back everything up to S3 in another region." |
| L5 | Names both gaps: "daily backups mean up to a day of data loss, and restoring takes hours during which we're down — so we'd want to know our RPO and RTO explicitly." |
| Staff+ | Tests the plan and prioritizes recovery: "a backup that's never been restored is a hypothesis — backups fail silently, and the classic case is the encryption key living only in the region that burned down. I'd run restore drills. Recovery also parallelizes across our separate stores, so RTO is the max rather than the sum, and I'd prioritize MySQL so questions and answers come back first and the ML feature store catches up after. On mitigating the data loss with synchronous cross-region replication — that's a much bigger claim than the intra-DC version, since it puts tens to hundreds of milliseconds on every critical write." |
Next: the whole design under interview conditions.