Why this matters: the naive broker — a map from topic to callback list, publish loops and calls each one — compiles, works in the demo, and is wrong in every way that matters: one slow subscriber stalls all of them, the publisher inherits strangers' latency, and a subscribe during publish is a race. The design that scores starts from a single structural claim and lets everything else follow from it. This lesson builds that chain.

Start from the operations, not the nouns

publish(topic, message)          -> returns FAST, no subscriber code runs
subscribe(topic, callback, cfg)  -> Subscription (mode, queue capacity)
unsubscribe(subscription)        -> clean stop, no half-delivered mess
ack(subscription, messageId)     -> at-least-once mode only

The annotation on publish is the design. Say it as a sentence before drawing anything: publish must not run subscriber code. A publish appends to each subscriber's queue and returns; separate delivery workers — one per subscription — drain those queues and invoke callbacks. Every guarantee the requirements demanded falls out of this one split: publishers never inherit subscriber latency, a slow subscriber is invisible to every other subscriber (its queue fills; nobody else's does), and per-subscriber ordering comes free because one worker drains one queue in sequence. This is the broker's equivalent of the allocator's header trick or the scheduler's neighbor check — the single move the whole round pivots on.

The ownership map

Broker
 └─ topic registry: name -> Topic (created on first use)
     └─ Topic
         ├─ publish sequencer (per-topic ordering point)
         └─ subscription list (copy-on-write)
             └─ Subscription (owns its whole delivery world)
                 ├─ bounded queue        (capacity from config;
                 │                        full -> drop oldest)
                 ├─ delivery worker      (drains in order, runs callback)
                 ├─ mode                 (at-most-once | at-least-once)
                 └─ pending/ack state    (at-least-once only)

The load-bearing decision is what the Subscription owns: its queue, its worker, its delivery mode, its config. Everything one subscriber's delivery needs lives inside its subscription — which means subscriptions cannot interfere with each other, and unsubscribe has one object to tear down. That's cohesion doing real work: the unit of ownership matches the unit of isolation. A design where the broker owns one shared delivery pool serving all queues re-couples what we just decoupled — now a stall in one callback occupies a shared worker and other subscribers feel it.

Ordering: one sequencer per topic, then it's free

The promise is per-topic publish order, delivered in order to each subscriber. Two links in that chain, each needing a mechanism you must name, not assume. First, "publish order" must exist: concurrent publishers on one topic need a single ordering point — a short per-topic critical section (or equivalent single sequencer) that assigns the message its position and appends to every subscription queue before releasing. Second, that order must survive to the callback: it does, automatically, because each subscription has one queue drained by one worker — FIFO in, FIFO out. Say the corollary too: two workers draining one subscriber's queue would break the promise, which is why "one queue, one worker" is an invariant and not a tuning choice. And across topics? No promise — so no shared sequencer, no global lock, and topics scale independently. Notice the economy: ordering costs exactly one small critical section per topic, placed precisely where the promise requires it and nowhere else. That's designing to the contract — the guarantees you didn't make are the parallelism you get to keep.

The bounded queue is the enqueue path's contract

Each subscription's queue has a fixed capacity from its config; when an append finds it full, the oldest queued message is dropped to make room. Implement it exactly and visibly — the drop is a deliberate, countable event (increment a per-subscription counter; observability lands in lesson 4), not a silent side effect. State the consequence plainly when presenting: a subscriber that can't keep up loses its oldest backlog first, and in at-most-once mode those messages are simply gone. Contracts you can state without flinching are the mark of a candidate who understands what they built.

The subscribe/unsubscribe race, named before it bites

While a publish iterates a topic's subscription list, another thread subscribes or unsubscribes. If the list is naively mutable, that's a concurrent-modification bug; worse, the semantics are undefined — does the brand-new subscriber get this message? The design must answer, and the clean answer is a copy-on-write subscription list: publish snapshots the list and delivers to exactly that snapshot; subscribe/unsubscribe swap in a new list. The boundary rule falls out crisply: a subscriber sees messages published after its subscription lands in the list — never a half-delivered message, never a duplicate from list mutation. Unsubscribe mirrors it: removal from the list stops new enqueues, then the worker drains or discards per mode and stops cleanly. Naming this race unprompted — and stating the boundary message's fate — is a top-tier signal, because it's the bug that ships in every naive broker.

At-least-once, designed rather than bolted on

For an at-least-once subscription, the worker delivers a message and holds it as pending until ack(messageId) arrives; a failed callback (or ack timeout) triggers redelivery. Own the consequence in the same breath: redelivery after a crash-after-processing-before-ack means the subscriber will eventually see a duplicate, so at-least-once callbacks must be idempotent — the broker guarantees delivery, not exactly-one delivery. At-most-once mode is the same worker minus tracking: invoke, move on; a failure is a loss, as contracted. The two modes share the queue-and-worker chassis and differ only in the post-delivery step — which is why mode lives on the Subscription, not the Broker.

The invariants, stated as a set

1. No subscriber code ever runs on a publisher's thread
2. One queue, one worker per subscription — per-subscriber
   order preserved by construction
3. Per-topic publish order assigned at one sequencing point
4. Enqueue on a full queue drops the oldest — counted, never silent
5. Publish delivers to a snapshot of the subscription list;
   a message is never half-delivered or list-race-duplicated
6. At-least-once: a message leaves pending only via ack;
   redelivery may duplicate — callbacks must tolerate it

What we rejected, and why

Callbacks invoked directly in publish — the natural first sketch, and the requirements kill it three ways: publisher inherits subscriber latency, one slow subscriber stalls the rest of the list, and "any thread may publish" turns every callback into a concurrency hazard for its author.

One shared queue per topic, workers per subscriber reading it — subscribers consume at different speeds; a shared queue forces either the slowest subscriber's pace on everyone or per-subscriber read positions in shared state, which is a fan-out queue built the hard way. Fan out at enqueue instead; per-subscriber queues make isolation structural.

A global delivery thread pool over all subscriptions — fewer threads, but a stalled callback now occupies a shared worker and unrelated subscribers queue behind it; isolation leaks away. If thread count ever matters, bound workers per subscription group — but don't pre-build that.