Use Cases and Requirements

In one line: the requirements list contains one item that no messaging queue chapter had — a retention policy — and it is the requirement that forces the whole storage design.

Use cases

Use case	What it means
Improved performance	Push-based distribution eliminates the need for recipients to poll for updates, which reduces latency and improves response times
Handling ingestion	Efficiently handles log ingestion. It buffers large data streams — such as user interactions — before delivering them to analytical systems. Meta's Scribe uses this pattern to route logs to specific consumers or archive them, managing enormous data volumes
Real-time monitoring	Raw or processed messages can feed multiple monitoring applications simultaneously for real-time system oversight
Replicating data	Distributes state changes asynchronously — a leader node pushing updates to followers or distributed caches. Also synchronizes state across multiple client views (web and mobile) in real-time apps like WhatsApp

Real-time monitoring is the use case that shows why pub-sub, specifically

Notice it says multiple monitoring applications simultaneously. That is the whole argument for pub-sub over a queue in one phrase.

The metrics stream feeds a dashboard, an alerting engine, a long-term archive, and an anomaly detector — all reading the same messages. With a queue, those four would compete: each message would go to exactly one of them, which is useless. With pub-sub, each gets a complete copy and none knows the others exist.

This is exactly the setup that building blocks described, and it explains why a pub-sub tier sits underneath most real monitoring pipelines.

Data replication via pub-sub is the same mechanism as database replication

"A leader node pushes updates to followers" is precisely what the databases material's leader-follower replication does — the replication log is a stream of state changes, and followers are subscribers to it.

Recognizing that pub-sub and replication logs are the same structure is a genuinely useful insight: it is why change-data-capture works, why Kafka is often used as a database's outbound log, and why the ordering guarantees in Lesson 7 matter so much. Apply state changes out of order and the replica diverges.

Using pub-sub in a chat application

"You're designing a real-time chat application with channels for different discussions. Map it onto pub-sub."

The mapping is direct, and it is a common interview warm-up:

Pub-sub concept	Chat application
Topic	A channel — one topic per channel, so `#engineering` and `#random` are separate topics
Publisher	Any member sending a message to that channel
Subscriber	Every member who has joined the channel — including their web and mobile clients as separate consumers
Delivery	A message published to the channel topic is read independently by each member's client, each tracking its own offset so it can resume after being offline

Two details worth volunteering: ordering must be strict within a channel (a reply must not precede the message it answers) but is meaningless across channels — which is exactly the per-partition ordering Lesson 7 provides. And offsets are what make "unread messages" work: a client that was offline resumes from its last offset rather than losing history.

Functional requirements

Requirement	Detail
Topic creation	Producers can create a topic to group related messages
Write messages	Producers can publish messages to a topic
Subscription	Consumers can subscribe to a topic to receive updates
Read messages	Consumers can read messages from subscribed topics
Retention policy	Users can specify a retention period for messages
Message deletion	The system automatically deletes messages after the retention period expires

The last two requirements are one idea, and it reshapes the storage design

Retention and automatic deletion are the pair that make this not a queue.

In a queue, a consumer's delete call removes the message — deletion is consumption-driven. Here, deletion is time-driven: messages expire on a schedule regardless of whether anyone read them, and no consumer's action removes anything.

Two consequences follow immediately:

Storage is sized by retention, not by backlog. You hold every message for the full period even if all subscribers consumed it in the first second. Retention becomes a capacity decision.
Readers cannot destroy data, so many readers can share one copy safely. That is what makes Lesson 6's single-copy-plus-offsets design possible — and it is precisely the property Lesson 4's design throws away.

Non-functional requirements

Requirement	Detail
Scalability	Handle increasing numbers of topics, producers, and consumers
Availability	Producers and consumers must be able to access the system at all times
Durability	Accepted messages must persist until delivered to all intended subscribers
Fault tolerance	Continue operating despite component failures
Concurrency	Safely handle simultaneous read and write operations

Note the three axes of scalability — they stress different components

"Topics, producers, and consumers" are three independent growth dimensions, and an interviewer may push on which breaks first:

More producers → write throughput. Handled by partitioning (Lesson 6), since partitions absorb writes in parallel.
More consumers → read throughput and metadata. Each consumer needs its own tracked offset, so a million consumers means a million offsets — which is why Lesson 10 puts them in a key-value store rather than a relational database.
More topics → cluster coordination. The cluster manager tracks which broker holds which topic, so topic count grows its state.

Saying "it scales horizontally" addresses none of these specifically. Naming which tier each axis stresses is the senior answer, and it mirrors the two-axis discussion in the messaging queue chapter.

Durability here is phrased more weakly than it looks

"Persist until delivered to all intended subscribers" sounds like the system tracks completion — but the functional requirements say deletion is driven by the retention period, not by delivery.

In practice these reconcile the way real systems do it: retention is set long enough that all reasonable subscribers will have consumed within the window. A subscriber offline for longer than the retention period loses those messages permanently.

That is an important operational fact and a good thing to raise: a consumer down for eight days against a seven-day retention comes back to a gap, and the system will not tell it. Monitoring consumer lag against the retention window is how you catch that before it happens.

Building blocks we will use

Building block	Used for
Database	Stores metadata, such as subscription details
Distributed messaging queue	Buffers messages sent by producers
Key-value store	Stores transient consumer state

Three chapters you have already built, composed rather than reinvented.

Key takeaway

Four use cases, all sharing one shape: several independent readers over the same stream. Six functional requirements, of which retention plus automatic deletion is the pair that separates this from a queue. And scalability runs on three axes — topics, producers, consumers — each stressing a different tier.

Interview signal by level

Level	What a strong answer sounds like
L4	"Publish, subscribe, read, and it should scale and be available."
L5	Flags retention: "messages expire on a retention policy rather than being deleted when consumed — that's what lets several subscribers read the same copy."
Staff+	Separates the scaling axes and names the operational trap: "topics, producers, and consumers grow independently — producers stress write throughput and want partitions, consumers stress offset metadata and want a key-value store, topics stress the cluster manager. And retention being time-driven means a subscriber offline longer than the window comes back to a silent gap, so I'd alert on consumer lag against retention."

Next: the six calls.