At-Least-Once and the Deduplication Boundary
In one line: the requirements lesson established that exactly-once delivery is unavailable, so this is where the guarantee actually gets built — and it is built at the consumer, not at the coordinator.
Where the duplicates come from
Four sources, and it is worth being able to enumerate them because each argues for the same fix.
Lease expiry. A worker submits a batch and dies before acknowledging. The lease expires, the batch is reissued, and it is submitted twice. This one is structural — it is the price of surviving worker death without liveness tracking.
Submission ambiguity. A worker submits and the acknowledgement is lost in the network. It cannot distinguish that from the submission being lost, so it retries. This is the Two Generals case arriving in practice.
Operator replay. Someone runs a replay, is unsure whether it worked, and runs it again. Extremely common during an incident.
The message was already partly processed. The original consumer may have written to a database and then failed on a downstream call. The message was dead-lettered, but part of its effect already landed.
The fourth is the one that catches people. Deduplicating replayed submissions does nothing about it, because the duplicate effect came from the original attempt. Only an idempotent consumer handles that case, which is the argument for where the boundary belongs.
The idempotency key
The key must identify the original message, not this attempt at it.
key = hash(source_topic, partition, offset) -- broker-native identity or hash(producer_id, sequence_number) -- producer-assigned or the business event id, where one exists -- best, if trustworthy
The rule that makes it work: the key must be identical across every replay of the same message, and different for genuinely different messages. So it cannot include the replay job id, the attempt number, or a timestamp — all of which change per attempt and would make every replay look new.
A business event id is the best key where the producer assigns one reliably, because it survives the message being moved between topics or re-serialised. Broker coordinates are the reliable fallback.
Where to deduplicate
Two places, and they solve different problems.
| At the coordinator | At the consumer | |
|---|---|---|
| Catches | Double submission from lease expiry, lost acks, operator error | Everything, including partial processing by the original attempt |
| Cost | One key-value lookup per message | Every consumer must implement it |
| Ownership | The platform team | Hundreds of service teams |
| Sufficient alone? |
The honest answer is both, for different reasons.
Coordinator-side deduplication is cheap and prevents obvious waste — it stops a doubly-leased batch consuming replay capacity twice, and it makes an operator's accidental re-run harmless. It is a good citizen, and it is not a correctness guarantee, because it cannot see what the original attempt already did.
Consumer-side idempotency is the actual guarantee. It is also the expensive one organisationally, because it is a requirement placed on every team that owns a consumer. Naming that cost rather than hand-waving it is the senior move: "effectively exactly-once means every consumer team implements idempotency, and that is a platform-wide contract, not something the coordinator can provide."
The deduplication store
Coordinator-side, the mechanics are ordinary and two details matter.
It must be a conditional write, not read-then-write. Two workers processing the same batch concurrently will both read "not seen" and both submit. Use an atomic insert-if-absent — SETNX, a conditional put, a unique constraint. Whoever wins submits; the loser skips.
It needs a TTL, and the TTL is a policy decision. Keys cannot be kept forever.
Retention too short -> a legitimate late duplicate slips through
Retention too long -> unbounded storage growth
Reasonable: longer than the maximum plausible replay window,
e.g. 7-30 days
From the estimation lesson: roughly 700MB per five-million-message replay at 128 bytes a key. With a few weeks of TTL and several concurrent replays that is tens of gigabytes — cheap, and worth bounding deliberately rather than discovering.
What idempotency means at the consumer
Worth being concrete, because "make it idempotent" is often said and rarely specified.
Conditional writes. INSERT ... ON CONFLICT DO NOTHING keyed on the event id, or an update guarded by a version or sequence number.
Upsert semantics. Applying the same state twice produces the same result. Naturally idempotent for last-write-wins data.
A processed-events table written in the same transaction as the effect. The strongest form: the record that a message was handled and the effect of handling it commit together, so there is no window where one exists without the other.
Natural idempotency. Some operations are inherently safe — setting a status to shipped twice is fine. Incrementing a counter twice is not, which is why "add ten dollars" has to become "set balance to X, if version is Y".
Key takeaway
Duplicates arrive from lease expiry, lost acknowledgements, operator re-runs, and — the one deduplication cannot fix — partial processing by the original attempt. Key on the original message's identity, from broker coordinates rather than payload content so a header mutation cannot change it. Deduplicate in both places for different reasons: coordinator-side is cheap and prevents waste but is not a guarantee, and consumer-side idempotency is the actual guarantee and a platform-wide contract worth naming as a cost. Use a conditional write rather than read-then-write, and give the store a TTL longer than the maximum plausible replay window.
Next: filtering, and the provenance that makes a replay auditable.