Cheat Sheet
Key takeaway
Distributed systems are hard for exactly one reason: partial failure. Some parts work, some don't, and the working parts can't tell which. RPC hides the network, consistency models specify what a read may return, and failure models specify how a node may break — every mechanism below exists to stay correct while "did it work?" is unanswered.
Key terms
| Term | One line |
|---|---|
| Abstraction | Hiding detail to reason about the goal, not the mechanism |
| Leaky abstraction | One whose hidden details reassert themselves under load or failure |
| End-to-end argument | Guarantees you truly need must be re-established at the endpoints |
| Marshaling | Converting parameters to a standard wire format (unmarshaling = the reverse) |
| Stub | Local proxy that makes a remote call look like a local one |
| IDL | Schema both sides compile stubs from (.proto, .thrift) |
| Idempotent | Doing it N times equals doing it once |
| Deadline | Absolute "give up at" instant, propagated down the call chain |
| Partial failure | Some components failed, and healthy ones can't tell which |
| Split brain | Two nodes simultaneously believe they own the same data |
| Fencing token | Monotonic epoch that lets storage reject a stale leader's writes |
| Gray failure | System reports healthy; users experience broken |
| Quorum | Overlapping read/write sets — r + w > n |
The eight fallacies
| # | Fallacy | Design force |
|---|---|---|
| 1 | The network is reliable | Timeouts, retries, idempotency, breakers |
| 2 | Latency is zero | Batch, colocate, cache, go async |
| 3 | Bandwidth is infinite | Compress, paginate, binary encodings |
| 4 | The network is secure | TLS everywhere, authn/authz per hop |
| 5 | Topology doesn't change | Service discovery, no hardcoded IPs |
| 6 | There is one administrator | Versioned APIs, backward compatibility |
| 7 | Transport cost is zero | Efficient codecs, locality-aware routing |
| 8 | The network is homogeneous | IDLs, schemas, explicit contracts |
Latency numbers worth knowing
| Operation | Rough time |
|---|---|
| L1 cache reference | ~1 ns |
| Main memory reference | ~100 ns |
| SSD random read | ~100 us |
| Round trip, same datacenter | ~500 us |
| Disk seek (spinning) | ~10 ms |
| Round trip, cross-continent | ~150 ms |
RPC in one picture
Runtime owns transmission, retransmission, acknowledgment, encryption. That retransmission is why delivery semantics are your problem.
Delivery semantics
| Semantic | Implementation | Use for |
|---|---|---|
| At-most-once | Send once, never retry | Metrics, telemetry, cache warming |
| At-least-once | Retry until acked | Almost everything — needs idempotency |
| Exactly-once (effective) | At-least-once + server dedup | Payments, orders, inventory |
Exactly-once delivery does not exist. Exactly-once processing does.
Idempotency key rules: client generates it · claim + effect in one transaction · store the response, not a flag.
Communication styles
| Style | Sync/Async | Reach for it when |
|---|---|---|
| gRPC | Sync | Internal service-to-service at volume; typed contract, streaming |
| REST | Sync | Public APIs, browsers, partners, cacheable reads |
| GraphQL | Sync | Many diverse clients over one rich graph |
| Messaging | Async | Fan-out, background work, spike absorption, replay |
Each synchronous dependency multiplies into your availability: 4 × 99.9% = 99.6%.
Consistency spectrum
WEAKEST STRONGEST Eventual → Causal → Sequential → Strict (linearizable)
| Model | Guarantees | Example |
|---|---|---|
| Eventual | Converges once writes stop; stale reads allowed | DNS, Cassandra, view counts |
| Causal | Dependent operations ordered; unrelated ones aren't | Comment threads (reply after parent) |
| Sequential | Each client's program order, one global interleaving; no real-time | Social feed per friend |
| Strict / linearizable | Read returns the latest write, immediately | Password change, account balance |
ACID C = the database's own invariants. CAP C = replicas agree. Unrelated.
Linearizable = recency, single objects. Serializable = isolation, transactions. Strict serializable = both.
Session guarantees (per client — cheap)
| Guarantee | Prevents |
|---|---|
| Read-your-writes | "My edit didn't save" |
| Monotonic reads | Content disappearing on refresh |
| Monotonic writes | Rename landing before create |
| Writes-follow-reads | Reply visible before its parent |
Implement with a version watermark: returned on write, echoed on read, keyed to the user not the connection.
CAP and PACELC
if (Partition) then Availability or Consistency Else then Latency or Consistency
- P is not optional. The real choice during a partition is CP or AP.
- CP when a wrong answer beats no answer being false: ledgers, credentials, inventory, blocks.
- AP when uptime wins: feeds, carts, DNS, catalogs, metrics.
- The
ELSEbranch governs your p99 every single day — a strong read costs a quorum round trip. r + w > nis what makes read and write sets intersect. Latency = the slowest node in the quorum.
Failure models
EASY ────────────────────────────────────────────── HARD Fail-stop → Crash → Omission → Temporal → Byzantine
| Model | Node behavior | Mitigation |
|---|---|---|
| Fail-stop | Halts, detectably | Remove from pool |
| Crash | Halts silently | Heartbeats + timeouts |
| Omission | Drops some messages | Acks, retries, per-node success rates |
| Temporal | Correct but too late | Deadlines, hedging, load shedding |
| Byzantine | Arbitrary / wrong / lying | Checksums, signatures, 3f+1 quorums |
Replicas to tolerate f failures: 2f + 1 crash, 3f + 1 Byzantine.
🔴 Writing to two places
Ordering doesn't help — it only picks which inconsistency you get.
| Two-phase commit | Saga | |
|---|---|---|
| Guarantee | Atomic | Eventually consistent — visibly partial in between |
| Locks | Held across services until the decision | None cross services |
| Coordinator dies mid-protocol | Participants block, holding locks | Resume the workflow; nothing stuck |
| Third-party participant | Impossible | Fine, if compensable |
| Reach for it | Inside one database/cluster | Across services — nearly always |
Why 2PC blocks: between voting and hearing the decision a participant can't resolve alone — it doesn't know how the others voted. So the coordinator is a single point of failure holding everyone's locks.
Saga caveats: a compensation is not a rollback (you can't un-send an email — you issue a correction, which is a new business event); compensations can fail, so orchestration must be durable, not in-memory; and every step must be idempotent.
Distributed atomicity is available and you usually shouldn't buy it.
Quick decision cues
- Caller needs the answer to respond → synchronous; otherwise → queue it
- Internal and high-volume → gRPC; browser or partner-facing → REST
- Retrying a write → it needs an idempotency key
- Multi-hop call chain → propagated deadlines, not per-hop timeouts
- Dependency is down, not slow → circuit breaker, fail fast
- "User can't see their own change" → read-your-writes, not strong consistency
- Stale read is merely cosmetic → eventual
- Stale read is a safety or money bug → linearizable
- Node is slow, not dead → shed and fence, don't evict
- Dashboards green, users angry → gray failure; measure client-side
Common gotchas
- Retries without jitter rebuild the thundering herd. Exponential backoff plus full jitter, always.
- Retries at multiple layers multiply. 3 layers × 3 retries = 27 requests per logical call. Retry at one layer.
- Adding read replicas silently breaks read-your-writes. Works in staging (no lag), fails in production.
- Health checks only detect crashes. They are blind to omission and temporal failures — the ones that hurt.
- Treating slow as dead causes split brain. The zombie leader returns and writes. Use fencing tokens.
- Renumbering a protobuf field breaks every client that hasn't redeployed. Add fields; never renumber.
- "We chose AP" as a whole-system label is a red flag. It's a per-data-type, per-operation decision.
- Assuming exactly-once delivery exists. It doesn't — build exactly-once processing.
Interview soundbites
- "Exactly-once delivery isn't achievable — at-least-once plus server-side dedup, with the key claimed in the same transaction as the effect."
- "CAP only tells you what happens during a partition. PACELC's else branch is what I pay for every day, and it's where my p99 goes."
- "That's read-your-writes, not strong consistency — a version watermark fixes it without a quorum round trip on every read."
- "These fields don't need the same guarantee. Avatar can be stale; a block list can't. One system, two consistency paths."
- "Slow isn't dead. If I evict an overloaded node I move its traffic onto its struggling peers — I'd shed load and fence the leader instead."
- "Each synchronous dependency multiplies into my availability, so I'm keeping the critical path to two calls."
Work the Interview Walkthrough section for these ideas applied end to end, and the Concept Drills for the rapid-fire versions.