Cheat Sheet

The one-line summary

Pub-sub delivers one message to every subscriber. A topic is an immutable, append-only log split into partitions across brokers; each partition is a series of segments addressed by offset. Every consumer keeps its own offset, so all read one copy independently. Deletion is retention-driven, not consumption-driven.

Queue vs pub-sub

Decoupling = three independences: immune to slow consumers · immune to consumer count (publish is O(1) in fan-out) · immune to consumer failure.

Use cases

Improved performance (push, no polling) · handling ingestion (log buffering — Meta's Scribe) · real-time monitoring (several apps read the same stream) · replicating data (leader → followers/caches; multi-device state sync).

Requirements

Functional: topic creation · write messages · subscription · read messages · retention policy · automatic deletion on expiry.

Non-functional: scalability · availability · durability · fault tolerance · concurrency.

Three scaling axes, three tiers: more producers → write throughput (partitions) · more consumers → offset metadata (key-value store) · more topics → coordination (cluster manager).

API

create(topic_ID, topic_name)
write(topic_ID, message)                  -> max 1 MB
write(topic_ID, partition_ID, message)    -> strict ordering
read(topic_ID)
subscribe(topic_ID)
unsubscribe(topic_ID)
delete_topic(topic_ID)

There is no delete_message. A consumer deleting a message would destroy other subscribers' data. That absence is what makes a shared single copy safe.

1 MB cap → messages stay small → append-only local storage works. Large payloads go to blob storage with a reference in the message.

First design (queues) — and why it fails

Components: topic queue · relational DB (subscriptions) · message director · per-consumer queue · subscriber.

Failure 1 — storage amplification. One copy per subscriber:

1 KB post x 10,000,000 followers = 10 GB written, per post
Publisher cost = O(N), not O(1)   <- wrong complexity class

Failure 2 — the obvious repair. Share one queue + reference count → head-of-line blocking: a queue serves from the head, so if 9 of 10 have consumed the head message, the tenth being slow freezes the other nine.

Both come from reusing the queue abstraction. The fix needs: one copy · independent reads at arbitrary positions · per-consumer position. That is an append-only log. (Analogy: Linux hard links — one copy, many references, delete at zero.)

Second design — components

Both managers are control plane — nothing flows through them. Data path is producer → broker → consumer, three hops.

Two acknowledgments, two guarantees: producer-side ack = durably stored (durability boundary). Consumer-side ack = processed up to here (offset commit).

Retention: default 7 days; banking may need weeks, analytics may discard immediately. It is simultaneously a capacity, compliance, and recovery-window decision.

Storage hierarchy

Topic (persistent, immutable sequence)
  └── Partition 0, 1, 2 ...        <- exists because DISK I/O caps one broker
        └── Segment files          <- retention = delete oldest segment, O(1)
              └── offset 0, 1, 2, 3 ...

• Partition count = maximum consumer parallelism. Over-provision; adding later breaks per-key ordering.
• Immutability makes readers cursors → independent consumption, no locking on the read path.

Placement and ordering

• Default: weighted round robin → even load, no ordering at all.
• partition_ID supplied → one log → strict ordering.
• Weighted, not plain, because brokers differ in speed and load — uniform distribution is only right when the targets are uniform.
• Only the client knows which messages are related, which is why it chooses. Kafka hashes a message key instead — same guarantee, less leaked.
• Cost: keyed writes bypass load balancing → hot partition. Fix by higher-cardinality key, or split the key and lose order across splits.

Storage: local disk, not S3

• Blob stores are not optimized for small reads/writes; geo-replication makes it worse.
• Local append-based writing → contiguous tracks and sectors, what disks are tuned for; contiguous regions cache well.
• Same lesson as Redis pipelining: when work per operation is tiny, per-operation overhead is the system.
• Trade: storage bounded by broker disk · durability is ours · storage coupled to compute → tiered storage for cold segments.
• Partitions spread across brokers contains failure; it does not survive it. That is replication's job.
• Round robin needs a metadata mapping (logical index → server/partition), because placement can't be recomputed.

Replication

• Three replicas per partition, on different brokers. Leader-follower; cluster manager elects on failure.
• Leadership is per partition, not per broker → write load spreads; a broker loss costs one partition's leadership.
• Three because two leaves no redundancy during repair; three also permits quorum (majority of 3 = 2).
• Spread across racks/zones — three brokers in one rack share a failure domain.
• Failover hazards: a lagging promoted replica drops acked messages silently, and can invalidate consumer offsets. Fix with a high-water mark — read only up to what all in-sync replicas have.

Offsets

Key-value store
  (consumer_B, topicA:p0) -> 2
  (consumer_A, topicA:p0) -> 6

Why key-value, not relational: one offset per consumer per partition, overwritten constantly, fetched by exact key, no joins. Subscriptions stay relational (low volume, queried relationally). Split metadata by access pattern.

Commit order decides semantics:

No ordering gives exactly-once. Get it from idempotency, or from an atomic commit-plus-side-effect.

Replay: set the offset backwards → reprocess history. Bounded by retention.

Push or pull is per consumer (stored in the DB): push for real-time (chat, dashboards), pull for batch. Pull keeps backpressure with the consumer.

Hazard: a consumer slower than retention has an offset before the log start — data lost, usually silently. Alert on lag against the retention window, and treat offset-before-start as a hard error, never an auto-seek.

The five sentences to have ready

1. "A queue message is consumed once; a pub-sub message is read by many, so retention drives deletion, not reading."
2. "Copy-per-subscriber is 10 GB for a 1 KB post to 10M followers — and O(N) publish cost is the wrong complexity class."
3. "Sharing a queue with a refcount gives head-of-line blocking: one slow subscriber freezes everyone."
4. "Immutable log + per-reader offset = independent consumption. That's the whole answer."
5. "Partitioning by key trades guaranteed load balance for guaranteed ordering."