Cheat Sheet

Key takeaway

Distributed systems are hard for exactly one reason: partial failure. Some parts work, some don't, and the working parts can't tell which. RPC hides the network, consistency models specify what a read may return, and failure models specify how a node may break — every mechanism below exists to stay correct while "did it work?" is unanswered.

Key terms

Term	One line
Abstraction	Hiding detail to reason about the goal, not the mechanism
Leaky abstraction	One whose hidden details reassert themselves under load or failure
End-to-end argument	Guarantees you truly need must be re-established at the endpoints
Marshaling	Converting parameters to a standard wire format (unmarshaling = the reverse)
Stub	Local proxy that makes a remote call look like a local one
IDL	Schema both sides compile stubs from (`.proto`, `.thrift`)
Idempotent	Doing it N times equals doing it once
Deadline	Absolute "give up at" instant, propagated down the call chain
Partial failure	Some components failed, and healthy ones can't tell which
Split brain	Two nodes simultaneously believe they own the same data
Fencing token	Monotonic epoch that lets storage reject a stale leader's writes
Gray failure	System reports healthy; users experience broken
Quorum	Overlapping read/write sets — `r + w > n`

The eight fallacies

#	Fallacy	Design force
1	The network is reliable	Timeouts, retries, idempotency, breakers
2	Latency is zero	Batch, colocate, cache, go async
3	Bandwidth is infinite	Compress, paginate, binary encodings
4	The network is secure	TLS everywhere, authn/authz per hop
5	Topology doesn't change	Service discovery, no hardcoded IPs
6	There is one administrator	Versioned APIs, backward compatibility
7	Transport cost is zero	Efficient codecs, locality-aware routing
8	The network is homogeneous	IDLs, schemas, explicit contracts

Latency numbers worth knowing

Operation	Rough time
L1 cache reference	~1 ns
Main memory reference	~100 ns
SSD random read	~100 us
Round trip, same datacenter	~500 us
Disk seek (spinning)	~10 ms
Round trip, cross-continent	~150 ms

RPC in one picture

Runtime owns transmission, retransmission, acknowledgment, encryption. That retransmission is why delivery semantics are your problem.

Delivery semantics

Semantic	Implementation	Use for
At-most-once	Send once, never retry	Metrics, telemetry, cache warming
At-least-once	Retry until acked	Almost everything — needs idempotency
Exactly-once (effective)	At-least-once + server dedup	Payments, orders, inventory

Exactly-once delivery does not exist. Exactly-once processing does.

Idempotency key rules: client generates it · claim + effect in one transaction · store the response, not a flag.

Communication styles

Style	Sync/Async	Reach for it when
gRPC	Sync	Internal service-to-service at volume; typed contract, streaming
REST	Sync	Public APIs, browsers, partners, cacheable reads
GraphQL	Sync	Many diverse clients over one rich graph
Messaging	Async	Fan-out, background work, spike absorption, replay

Each synchronous dependency multiplies into your availability: 4 × 99.9% = 99.6%.

Consistency spectrum

WEAKEST                                                    STRONGEST
Eventual  →  Causal  →  Sequential  →  Strict (linearizable)

Model	Guarantees	Example
Eventual	Converges once writes stop; stale reads allowed	DNS, Cassandra, view counts
Causal	Dependent operations ordered; unrelated ones aren't	Comment threads (reply after parent)
Sequential	Each client's program order, one global interleaving; no real-time	Social feed per friend
Strict / linearizable	Read returns the latest write, immediately	Password change, account balance

ACID C = the database's own invariants. CAP C = replicas agree. Unrelated.

Linearizable = recency, single objects. Serializable = isolation, transactions. Strict serializable = both.

Session guarantees (per client — cheap)

Guarantee	Prevents
Read-your-writes	"My edit didn't save"
Monotonic reads	Content disappearing on refresh
Monotonic writes	Rename landing before create
Writes-follow-reads	Reply visible before its parent

Implement with a version watermark: returned on write, echoed on read, keyed to the user not the connection.

CAP and PACELC

if (Partition)  then Availability or Consistency
Else            then Latency      or Consistency

P is not optional. The real choice during a partition is CP or AP.
CP when a wrong answer beats no answer being false: ledgers, credentials, inventory, blocks.
AP when uptime wins: feeds, carts, DNS, catalogs, metrics.
The ELSE branch governs your p99 every single day — a strong read costs a quorum round trip.
r + w > n is what makes read and write sets intersect. Latency = the slowest node in the quorum.

Failure models

EASY ────────────────────────────────────────────── HARD
Fail-stop → Crash → Omission → Temporal → Byzantine

Model	Node behavior	Mitigation
Fail-stop	Halts, detectably	Remove from pool
Crash	Halts silently	Heartbeats + timeouts
Omission	Drops some messages	Acks, retries, per-node success rates
Temporal	Correct but too late	Deadlines, hedging, load shedding
Byzantine	Arbitrary / wrong / lying	Checksums, signatures, 3f+1 quorums

Replicas to tolerate f failures: 2f + 1 crash, 3f + 1 Byzantine.

Quick decision cues

Caller needs the answer to respond → synchronous; otherwise → queue it
Internal and high-volume → gRPC; browser or partner-facing → REST
Retrying a write → it needs an idempotency key
Multi-hop call chain → propagated deadlines, not per-hop timeouts
Dependency is down, not slow → circuit breaker, fail fast
"User can't see their own change" → read-your-writes, not strong consistency
Stale read is merely cosmetic → eventual
Stale read is a safety or money bug → linearizable
Node is slow, not dead → shed and fence, don't evict
Dashboards green, users angry → gray failure; measure client-side

Common gotchas

Retries without jitter rebuild the thundering herd. Exponential backoff plus full jitter, always.
Retries at multiple layers multiply. 3 layers × 3 retries = 27 requests per logical call. Retry at one layer.
Adding read replicas silently breaks read-your-writes. Works in staging (no lag), fails in production.
Health checks only detect crashes. They are blind to omission and temporal failures — the ones that hurt.
Treating slow as dead causes split brain. The zombie leader returns and writes. Use fencing tokens.
Renumbering a protobuf field breaks every client that hasn't redeployed. Add fields; never renumber.
"We chose AP" as a whole-system label is a red flag. It's a per-data-type, per-operation decision.
Assuming exactly-once delivery exists. It doesn't — build exactly-once processing.

Work the Interview Walkthrough section for these ideas applied end to end, and the Concept Drills for the rapid-fire versions.