CAP, PACELC, and the Cost of Consistency

Why this matters: CAP is the most cited and most misused theorem in system design. Used precisely it makes your trade-offs legible; used loosely it produces confident nonsense like "we chose AP" about a system that has never seen a partition.

Key takeaway

CAP says only this: when a network partition occurs, you must choose between consistency and availability. It says nothing about the 99.9% of the time there is no partition. PACELC completes the picture: else — when running normally — you still trade latency against consistency, on every single request.

What CAP actually says

Three properties, of which a distributed system can guarantee at most two:

Property	Means	Violated when
Consistency	Every read sees the most recent write (linearizability)	A replica serves a stale value
Availability	Every request to a live node gets a non-error response	A node refuses to answer to protect correctness
Partition tolerance	The system keeps working when messages between nodes are lost	The system requires a perfect network

The decision rule is one question: during a partition, is a wrong answer worse than no answer? For an account balance, yes — refuse. For a shopping cart, no — accept the write and reconcile later, because a rejected add-to-cart is lost revenue and a duplicated cart item is a minor annoyance.

PACELC — the framing you should actually use

CAP describes a rare event. PACELC (Abadi) covers all the time:

if (Partition)  then choose Availability or Consistency
Else            then choose Latency      or Consistency

The ELSE half is the one that governs your p99 every day. Even with a perfectly healthy network, a linearizable read must confirm with a quorum before answering — and that confirmation is a round trip you cannot optimize away. Consistency is not free during partitions; it is not free the rest of the time either.

System	Partition behavior	Normal-operation default	PACELC
Cassandra	Stays available, serves stale	Favors low latency (tunable)	PA/EL
DynamoDB	Stays available	Eventual by default; strong reads opt-in	PA/EL
MongoDB	Primary-based; minority side rejects writes	Favors consistency	PC/EC
Spanner	Minority partition loses writes	Favors consistency, pays commit-wait latency	PC/EC
DNS	Always answers	Cached aggressively; stale by design	PA/EL

Why consistency costs latency — the mechanism

It is not a law of nature; it is a round-trip count. Strong consistency needs the read set and write set to overlap, so a reader is guaranteed to touch at least one node that saw the newest write.

r + w > n

With n = 3 replicas, w = 2 and r = 2 intersect on at least one node. A write returns only after 2 acknowledgments, a read only after 2 responses — so each operation's latency is set by the slowest node in its quorum, not the average. Add cross-region replicas and that slowest node is 150 ms away.

Lower w to 1 and writes get fast and highly available — but now r must be 3, so reads must reach every replica and a single slow node stalls them. The budget is fixed; you choose where to spend it. (The Key-Value Store chapter works this dial in depth.)

What the systems you use actually guarantee

Engineers routinely assume guarantees their storage does not provide. Worth knowing cold:

System	Real guarantee	The gotcha
Amazon S3	Strong read-after-write for objects (since 2020)	Object reads are strong; cross-region replication is still async
DynamoDB	Eventually consistent reads by default	Strong reads cost 2x the read units and only work in-region
Cassandra	Tunable per query via consistency level	QUORUM reads and writes together give strong; ONE gives neither
PostgreSQL + read replicas	Strong on the primary	Replicas lag — a read replica silently breaks read-your-writes
Spanner	Strict serializability globally	You pay TrueTime commit-wait on every write
Kafka	Per-partition ordering, at-least-once	No ordering across partitions — key your messages deliberately

Saying it well in an interview

Weak: "It's a distributed system so we pick AP."

Strong: "During a partition this data can't tolerate a stale read — a stale balance means an overdraft — so writes go CP: the minority side rejects them and we take the write unavailability. The feed is the opposite; it stays available on stale data. And on the ELSE side, keeping the ledger consistent costs a quorum round trip on every read, which is where the p99 budget goes."

The difference is not vocabulary. It is that the second answer names the data, the failure, the cost, and who pays it.