Fault Tolerance and the Write Buffer

In one line: the offline-driver mechanism here is the most genuinely distributed-systems idea in the chapter — the client becomes a replica. And the replication claim beside it gets a fundamental trade backwards.

Handling a driver who goes offline

How will we handle a driver's slow and disconnected network?

The driver's device can persist trip state locally at regular intervals. Pending requests and the most recent trip state are written to durable local storage so they survive application restarts. If the driver closes the application while some requests are still pending synchronization, the application reloads the pending requests and the last known state from local storage on restart. The trip resumes from the previously persisted state, and the pending requests are re-enqueued and synchronized with the server.

The phone becomes a replica, and that is the right model

This is more interesting than "cache things offline." The driver's device is holding authoritative state about a trip that is physically happening, and the server is temporarily unable to see it.

Consider what is actually true during a tunnel: the trip is progressing. The car is moving, the meter is running, the rider is aboard. The physical world does not pause because the network did. So the phone is not caching a server value — it is recording ground truth that the server has yet to learn.

That inverts the usual relationship. Normally the server is authoritative and the client displays a possibly-stale copy. Here, for the duration of the outage, the client is authoritative about what happened.

Which makes reconnection a merge, not a refresh:

• The phone replays pending events in order.
• The server applies them to the trip state machine, which must accept them late and idempotently (Lesson 7).
• Conflicts are possible — the rider may have cancelled during the outage, while the driver's phone believes the pickup happened.

That last case has no clean technical answer, and the resolution is a policy decision: whose version wins when a rider cancels and a driver simultaneously confirms pickup? Real systems generally favour the physical evidence — the location trace showing the car stopped at the pickup point — and escalate to support.

When a system controls physical actions, the edge holds truth the centre has not received yet. Designing for that means idempotent replay and an explicit conflict policy, not just retry.

What the design does not say: the rider's side

The mechanism is described entirely for drivers. But a rider in the same tunnel has the same problem, and it is worse in one respect.

A driver's phone knows what the driver did. A rider who loses connection loses their only channel for "where is my driver?" and "the trip is progressing normally." They cannot see the car moving on the map, and they cannot cancel.

The asymmetry is worth naming because it changes what each side should cache: the driver caches events to send; the rider caches the last known state to display, plus a clear indication that it is stale.

And there is a third party the design never mentions: what if the server thinks a trip is in progress and neither phone has spoken for ten minutes? The trip state machine needs a timeout path and an escalation, or trips leak into a permanently in-progress state.

The write buffer

One million drivers may push location updates approximately every four seconds. This generates sustained write pressure on the quadtree service and can degrade its performance. To absorb this load, location updates are first written to a Redis hash for fast, in-memory ingestion. The updates are then flushed to persistent storage in batches every 10 to 15 seconds.

Redis is the DriverLocationHT from Lesson 6, made concrete

Lesson 6 introduced a hash table sitting in front of the quadtree, absorbing updates in O(1) and letting the tree lag. This is that hash table, named: a Redis hash, keyed by driver ID.

The choice fits the access pattern exactly:

That last row is the one people miss. This data is unusually disposable. Losing the entire Redis contents costs at most four seconds of staleness, because every driver re-reports continuously. So durability guarantees you would fight for elsewhere are simply unnecessary — and that is what makes an in-memory store acceptable for data on the critical path.

Data that regenerates itself does not need to be durable. The same argument that building block made for rebuilding a quadtree rather than restoring it, applied to a live stream.

One million drivers here, three million everywhere else

The lesson says "one million drivers may push location updates." Lesson 3's assumptions say 3 million active drivers, and every other calculation uses that.

1M / 4 s =  250,000 updates/second
3M / 4 s =  750,000 updates/second

A 3x discrepancy on the chapter's most important rate. The 3 million figure is the one consistent with the stated assumptions and the bandwidth calculation, so carry that.

It does not change the conclusion — 250,000 writes per second also requires a buffer — but it is worth catching, because the whole argument for Redis is quantitative and a threefold error in the driving number would matter if you were sizing the cluster.

Batching converts random writes into sequential ones

"Flushed to persistent storage in batches every 10 to 15 seconds" is doing more work than it appears.

Applied individually, 750,000 updates per second are 750,000 random writes — each touching a different key, in a different place, with per-operation overhead.

Batched over 15 seconds, they become one bulk operation carrying up to 3 million entries, written sequentially. The difference is not incremental; random and sequential write throughput differ by orders of magnitude on any storage medium.

There is a second saving specific to this workload: a driver produces roughly four updates in a 15-second window, and only the last one matters for the persisted latest-position record. So the batch can collapse them, cutting the persisted volume by about 75% before it is written at all.

Buffering does not only defer work — it lets you discard work that has been superseded. That is the real win, and it only exists because the data is an overwrite rather than an append. Route traces, being append-only, cannot be collapsed this way — which is another reason Lesson 3's omitted 162 GB/day matters.

Replication

For availability, we need database replicas. We use the primary-secondary replication model. We have one primary database and a few secondary databases. We synchronously replicate data from primary to secondary databases. Whenever our primary database is down, we can use a secondary database as the primary one.

Synchronous replication is being credited for availability, and it costs availability

This is stated under a heading called "Fault tolerance" and reappears in Lesson 14's evaluation as the technique for both availability and consistency. It cannot be both, and the direction is backwards.

Synchronous replication means the primary does not acknowledge a write until secondaries have it. What that buys and costs:

So synchronous replication trades write availability for consistency and durability. That is the CAP trade, stated plainly, and claiming both sides of it is the error.

Now — is it the right choice here? For the data it applies to, largely yes, and the reasoning is Lesson 2's scoping:

In-progress trips in MySQL: yes. Both parties must agree on trip state, and losing a trip transition on failover means a rider and driver with divergent views. That is precisely the case worth paying write latency for.

Driver locations: absolutely not. Synchronously replicating 750,000 writes per second would be ruinous, and it is unnecessary because the data regenerates in four seconds. Cassandra's tunable consistency handles this — write at a low consistency level and accept the occasional loss.

So the accurate statement is: "synchronous replication for the trip state, because both parties must agree; asynchronous or tunable for locations, because they are disposable." The design already splits the stores along that line; the replication description just fails to split with it.

Name what each replication mode costs, and scope it to the data that needs it. Saying "synchronous replication gives us availability" is the single most common consistency error in system-design answers.

When the driver simply does not answer

Accept and decline are the easy cases. The common one is no response at all, and a design that only handles explicit answers leaves the rider waiting indefinitely.

Schedule the timeout at the moment you send the offer. A delayed message due at the end of the acceptance window either gets cancelled by an explicit answer or fires and advances to the next driver. The retry becomes a property of the system rather than something a rider's client has to poll for.

Note this pairs with the lock TTL rather than duplicating it: the lock expiry frees the driver for other matches, and the delay message drives the ride forward to a new candidate. Two clocks, two different jobs, and both are needed.

The general shape — a human-in-the-loop step with a bounded response window and an automatic next action — is exactly what durable workflow engines exist for, and it is reasonable to say you would reach for one rather than hand-rolling retry state.

Absorbing peak demand

Demand is genuinely spiky — a concert ending puts thousands of requests into one small area at once. Processing them synchronously means the matcher is the thing that fails under exactly the load it exists for.

Putting requests on a durable queue does two things: it absorbs the burst so matching runs at its own sustainable rate, and it means a matcher instance crashing mid-match loses no request, because the message was never acknowledged.

Next: moving the money.