Cheat Sheet

The one-line summary

A distributed messaging queue is a durable buffer between producers and consumers, spread across machines. Stateless frontends validate and deduplicate, a metadata service maps a queue to its owning host, and backend hosts store and replicate the messages. Delivery is at-least-once, so consumers must be idempotent.

Components and patterns

• Producers create and send · consumers receive and process · queues buffer in FIFO order · messages carry a payload plus metadata.
• Point-to-point (one-to-one): a second consumer splits the work. Task queues, workers.
• Publish/subscribe (one-to-many): a second subscriber duplicates the work. Event-driven fan-out.
• Request/reply (two-way, synchronous): immediate feedback; gives up the decoupling benefit.

Benefits: performance and scalability (async) · decoupling (operate, scale, fail independently) · fault tolerance (persistence, retries, DLQs) · rate limiting and priority.

Technologies: RabbitMQ (broker, AMQP, complex routing) · Kafka (high-throughput streaming, log aggregation) · Amazon SQS (managed).

Use cases share one signature: the work is expensive and the user doesn't need the result to continue — email dispatch, media post-processing, recommender systems.

A queue buys you time, not capacity. It absorbs a burst, not a sustained overload.

Best practices

• Idempotency: processing a message N times equals processing it once. Mandatory, because at-least-once guarantees duplicates. Implement via naturally idempotent operations, message-ID dedup (needs atomicity with the work), or idempotency keys.
• Monitoring/logging: monitoring detects bottlenecks and latency spikes; logging captures events and errors. Alert on message age, not queue depth — age maps to user impact.
• Error handling: retries (with exponential backoff and jitter, or you thundering-herd the dependency) · dead-letter queues · circuit breakers. Retries handle a blip; circuit breakers handle an outage.

Requirements

Functional: create queue · send · receive · delete message · delete queue.

Delete is separate from receive on purpose — that two-step protocol is what makes at-least-once possible.

Non-functional: durability · scalability · availability · performance.

Single-server queue fails all four: high latency (lock contention — throughput goes down as clients rise), low availability (SPOF, no replication), no durability, no scalability.

Ordering

Who assigns the sequence: the producer. Only the producer knows production order.

Three mechanisms:

1. Monotonic server counters — bottleneck under bursts; records arrival order as if it were production order. Worst.
2. Client timestamps (causality) — respects production time; breaks on clock skew across sessions.
3. Synchronized-clock timestamps + process ID — unique, globally ordered, lets the server wait for delayed messages. Most robust — the sequencer building block.

Sorting: online sorting as messages arrive. If a late message arrives after a newer one was delivered → route to a special queue; the client decides to consume or discard.

Costs of strictness:

• Latency — the time window for stragglers, paid by every message. Size from p99 arrival skew.
• Throughput — order caps consumer parallelism. Ordering domain = parallelism domain. Scope to a partition.

Concurrency: locking (contention grows with contenders) vs serialization (append to a buffer, one thread drains — lock-free, higher throughput). Break the single-thread ceiling with multiple independently-serialized queues.

Extraction: pull vs push

Long polling = push latency with pull semantics: the request blocks at the broker (SQS: up to 20 s) until a message arrives. Default choice.

Push needs prefetch/credit limits for flow control — which is pull, reimplemented. Kafka: pull. SQS: pull + long polling. RabbitMQ: push with prefetch.

High-level design

Load balancer → stateless frontends → metadata service → backend servers → queues.

Frontend (stateless): request validation · authn/authz · caching · dispatching · deduplication (hash key store) · usage collection.

Frontend dedup stops double enqueue. It is not exactly-once — the duplicate that matters happens after enqueue.

Metadata service: read-through cache, write-through on create/delete.

Metadata cluster organization:

• Small metadata → replicate the full copy to every server; load-balance across them.
• Frontend mapping → shard table on the frontends. One hop; every frontend must agree.
• Host-based mapping → shard table on each metadata host. Two hops; one place to update. Suits read-intensive.

Backend: two models

• Primary-secondary: one named owner per queue; frontend asks metadata who owns queue 101. Only way to get total order. Unavailable during election.
• Cluster of independent hosts: any host takes the write and replicates via a local queue→host table. No election, no hotspot, no total order. Leaderless.

Replication:

• Synchronous — ack after secondaries confirm. Consistent replicas; extra delay; partial/no availability during election.
• Asynchronous — ack then replicate. Lower latency; an acked message can vanish with the primary, silently.

Durability is not "we replicate." It is "synchronous to at least one replica before we acknowledge."

Message deletion

Stream → offsets. Work queue → visibility timeout.

The trap: timeout expires while the consumer is still working → a second worker duplicates the job, nothing errors, and it's self-amplifying under load. Size off p99.9 processing time; extend mid-processing via heartbeat.

Dead-letter queue holds: max retries exceeded (poison message) · queue no longer exists (coordination failure) · TTL expired (capacity failure). Different causes, different responses — tag the reason. It's an alerting surface, not a graveyard.

Evaluation

• Durability: replicate metadata and message data.
• Scalability: two independent axes — message volume (expand storage past ~80%) and queue volume (cluster manager adds servers; performance isolation against noisy neighbours).
• Availability: replication + load balancers routing around failures. Under primary-secondary, bounded by election time.
• Performance: caching, replication, partitioning. Best-effort ordering by default; time-window sorting when strictness is required.

The five sentences to have ready

1. "Delete is separate from receive — that's what makes at-least-once possible."
2. "At-least-once guarantees duplicates, so idempotency lives in the consumer and is mandatory."
3. "Only the producer knows production order — a server-side counter records arrival order as if it were truth."
4. "Ordering domain = parallelism domain. Scope strict order to a partition."
5. "Async replication here means an acked message can vanish, silently, with the producer believing it succeeded."