Idempotency, Monitoring, and Error Handling

In one line: these are labelled "best practices," which undersells them. In a system that retries, idempotency is not a nice-to-have — it is the property that makes retrying safe at all.

Message idempotency

Idempotency ensures that processing a message multiple times yields the same result as a single execution.

This is critical in distributed systems where network issues or failures trigger retries. Idempotent design prevents duplicate actions and data corruption, simplifying error recovery.

At-least-once delivery makes duplicates inevitable, not unlikely

Point-to-point queues give at-least-once semantics — the queue retries anything unacknowledged. Consider where that bites: a consumer processes a message successfully, then dies before its acknowledgment reaches the queue. The queue has no way to distinguish that from a consumer that died before doing the work. So it redelivers.

The work happens twice, and no amount of care in the queue prevents it. This is the RPC failure-semantics problem from the Foundations module in its purest form: the sender cannot tell "the request never arrived" from "the response never came back."

The consequence is stark. If the handler charges a card, "at-least-once" means charging twice. Idempotency is what converts a duplicate delivery from a correctness bug into a no-op, and it must live in the consumer, because the queue cannot provide it.

How to actually make a handler idempotent

Three patterns, in increasing generality:

1. Naturally idempotent operations. SET status = 'shipped' is safe to repeat; INCREMENT retry_count is not. Prefer absolute writes over relative ones where you can — the same reasoning behind PUT versus a delta in that building block.
2. Deduplication by message ID. Record processed message IDs in a store with a TTL, and skip anything already seen. Note this is only as strong as the atomicity between "record the ID" and "do the work" — if they aren't in one transaction, you have moved the race rather than removed it.
3. Idempotency keys. The producer stamps a business-level key (order-1234-charge), and the downstream service treats a repeat of that key as the same operation and returns the original result. This is how payment APIs do it, and it survives across service boundaries where message IDs do not.

In an interview, saying "the consumer must be idempotent" is good; saying how is better.

Getting the message into the queue at all

Idempotency handles a message delivered twice. There is an earlier failure that handles neither, and it is the one interviewers reach for most often: the producer has to update its own database and publish to the queue, and those are two systems.

This is the dual-write problem, and no ordering of the two operations fixes it. Write to the database first and a crash loses the message silently; publish first and a crash leaves a message describing something that never happened. There is no atomic operation spanning two independent systems, and a distributed transaction across them is both unavailable in practice and undesirable.

The pattern that resolves it makes the two writes one write:

The transactional outbox: the service writes the business record and a row describing the message-to-be-sent in the same local transaction, which the database makes atomic. A separate relay process then reads unsent outbox rows and publishes them, marking each sent afterwards.

The crash windows all become harmless. Crash before commit and neither the record nor the message exists. Crash after commit but before the relay runs and the message is still sitting in the outbox, waiting. Crash after publishing but before marking the row sent and the relay republishes it — a duplicate, which is exactly what the idempotent consumer above already handles.

Which is the point worth making explicitly: the outbox converts a lost message into a duplicate message, and duplicates are a problem we have already solved. Trading an unrecoverable failure for a recoverable one is most of what reliability engineering is.

Two ways to run the relay, and the difference matters:

Start with polling. It is a query and an update, and it is correct; the change-log version is an optimization for when the poll interval starts showing up in user-visible latency.

Monitoring and logging

Monitoring and logging provide visibility into message flow and are essential for system health.

Together they enable proactive intervention and faster troubleshooting.

The two queue metrics worth alerting on

Generic advice like "monitor the queue" is weak. Two signals carry almost all the information:

Queue depth — how many messages are waiting. Rising depth means consumers are behind producers.

Consumer lag / message age — how long the oldest unprocessed message has been waiting. This is the one that maps to user-visible impact, because it answers how stale is the work.

Depth alone misleads: a depth of 10,000 on a queue that drains in two seconds is fine; a depth of 50 on a queue where the oldest message is an hour old is an incident. Age is the symptom; depth is the cause. The server-side monitoring chapter made the same argument about alerting on what users feel rather than on internal counters.

Error handling

Robust error handling maintains system stability during failures. Three common strategies:

Retries need backoff, or they become the outage

An unqualified "we retry" is a trap. If a downstream service is failing because it is overloaded, immediate retries multiply the load against it — every failure generates another request, and the service can never recover enough to serve anyone.

The fix is exponential backoff with jitter: wait progressively longer between attempts, and randomize the wait so all consumers do not retry in lockstep. Without jitter you get a thundering herd — thousands of clients that failed at the same moment retrying at the same moment.

That is precisely what the circuit breaker in the third row exists to prevent at the next level up: after enough failures it stops calling entirely for a period, giving the dependency room to recover. Retries handle a blip; circuit breakers handle a genuine outage.

A dead-letter queue is an alerting surface, not a graveyard

Messages land in the DLQ because they failed repeatedly. If nobody looks at it, you have built a place for data to go and quietly die — arguably worse than dropping messages, because the failure is now invisible instead of loud.

Treat any growth in DLQ depth as an alert. The messages there are exactly the ones telling you something is systematically wrong: a poison-pill payload your parser rejects, a schema change nobody coordinated, a downstream contract that moved.

Lesson 9 covers the full set of conditions that route a message to a DLQ.

What happens without a queue at all?

"How would a system behave if producers were much faster than consumers and no messaging queue existed?"

Without a buffer, the producer calls the consumer directly and synchronously, so the mismatch surfaces immediately:

• The producer blocks on every call, inheriting the consumer's latency and its saturation.
• The consumer's inbound connections and thread pool saturate, and new requests are rejected or time out — work is lost, not delayed.
• Timeouts propagate back to the producer's own callers, so the producer starts failing too. The Foundations module's cascading-failure shape, exactly.
• There is nowhere to absorb a spike, so the system's capacity is permanently sized for its peak rather than its average — which is expensive and still fails when peak is exceeded.

The queue converts all of this from lost work into delayed work. That is the trade, and it is a good one right up until the backlog stops draining.

Interview signal by level

Next: what we're actually building.