Task Idempotency

In one line: the scheduler will run some tasks twice. That is not a bug to fix — it follows from retrying, which follows from fault tolerance. Idempotency is what makes it survivable.

The worked failure

A has 20 dollars and sends 10 dollars to B, who has 0.

Without an idempotence key
--------------------------
1. A: "send 10 dollars to B"          A=20  B=0
2. Server begins adding 10 to B       ... does not complete successfully
3. Server sends an error to A
4. A retries the transaction
5. Server begins adding 10 to B       <- may now apply a SECOND time

The failure is that 'it failed' and 'the response was lost' look identical

Step 3 is where the problem lives. A received an error — but an error tells A that something went wrong, not what. Three very different states produce the same error:

• The transfer never started. Retrying is correct.
• The transfer completed, but the acknowledgment was lost in the network. Retrying double-pays.
• The transfer partially applied. Retrying does something unpredictable.

From A's side these are indistinguishable, and no amount of care on A's part resolves it. This is the RPC failure-semantics problem from the Foundations module, in its most expensive form: the sender cannot tell "the request never arrived" from "the response never came back."

Notice this makes retrying unsafe and unavoidable at the same time. Lesson 2 required fault tolerance, which requires retries — so the scheduler must retry, and retrying can corrupt data. The only escape is to change the task, not the scheduler.

Same conclusion message-queue design reached about at-least-once delivery: duplicates are structural, so the handler must tolerate them.

The fix: an idempotence key

With an idempotence key
-----------------------
1. A: "send 10 dollars to B" + idempotence key K    A=20  B=0
2. Server begins adding 10 to B      ... does not complete successfully
3. Server sends an error to A
4. A retries, attaching the SAME key K
5. Server checks K:
     - already applied?  -> return the original result, change nothing
     - not applied?      -> apply it, record K
                                                     A=10  B=10

To prevent data corruption, tasks must be idempotent. An idempotent task produces the same result regardless of how many times it is executed.

The key is supplied by the client, and it has to be

Note what step 1 requires: the client attaches the key before the first attempt, and reuses the same key on retry.

That is not incidental. The key is what makes two requests recognizable as the same logical operation — and only the client knows that. To the server, two identical transfer requests could be one retry or two genuine transfers of 10 dollars each, and it cannot tell the difference from the payload.

This is the same conclusion as pub-sub's partition key and message-queue design's sequence numbers: the semantic relationship between operations exists only in the client's head. The system can only preserve what the client tells it.

So idempotency is a contract, not a feature the scheduler provides. It requires cooperation, and a client that generates a fresh key per attempt gets no protection at all.

Recording the key and doing the work must be atomic, or you have moved the race

The mechanism in step 5 — check the key, do the work, record the key — has a gap. If the server records the key and then crashes before completing the work, the retry sees "already applied" and skips work that never happened. If it does the work and crashes before recording, the retry repeats it.

So the check-and-record must be atomic with the side effect, typically by writing the key inside the same database transaction as the balance update. Anything less relocates the race rather than removing it.

The Distributed Messaging Queue chapter made this exact point about message-ID deduplication, and it is the detail that separates someone who has implemented idempotency from someone who has read about it.

Where the side effect is not in a transactional store — sending an email, calling a third-party API — you cannot get atomicity, and you fall back to making the operation naturally idempotent or accepting rare duplicates.

The task that kills every worker it touches

Retrying is the right default and it has a pathological case: a task that fails because of itself rather than because of a transient condition.

A poison pill is deterministic, so no amount of retrying helps — and because the scheduler cannot distinguish "the network blipped" from "this input is malformed", a naive retry policy will feed it to worker after worker.

The defence is bounded attempts followed by a dead-letter queue: after N failures the task is set aside with its error rather than retried, and a human or a repair job looks at it. Two properties matter — the pool stays healthy, and the failure becomes visible rather than being an endlessly-retrying job nobody notices.

Worth pairing with the retry policy explicitly: exponential backoff handles transient failure; a retry cap handles deterministic failure. A design with backoff but no cap has only solved half the problem.

The general pattern

Consider uploading a video. To ensure idempotency, the developer assigns a unique identifier (e.g. the file name) to the video. If the scheduler retries the upload, the system detects the existing identifier and either overwrites the file or ignores the duplicate. This ensures the final state remains consistent even after multiple execution attempts.

Naturally idempotent operations need no key at all

The video example is subtly different from the money transfer, and the difference is worth naming.

Writing a file to a fixed path is naturally idempotent — do it twice and the result is identical, because the operation is an absolute assignment rather than a relative change. No key, no deduplication table, no atomicity concern.

Adding 10 dollars is relative, so repeating it changes the outcome. That is why it needs the key.

The general rule, which message-queue design also reached: prefer absolute writes over relative ones. SET balance = 10 is safe to repeat; balance = balance + 10 is not. Where you can express the work as "make the world look like this" rather than "apply this change," idempotency comes free.

Three levels of solution, worth being able to rank:

1. Naturally idempotent — restructure the operation. Best.
2. Idempotence key — deduplicate explicitly, atomically with the side effect. Necessary when the operation is inherently relative.
3. Accept duplicates — when they are genuinely harmless.

Why idempotency matters even with reliable acknowledgments

"Why does idempotency matter even when we have robust machine acknowledgments in place?"

Because no acknowledgment protocol closes the gap. Making acks more reliable shrinks the window in which one is lost; it never eliminates it, since the ack itself can always be the message that fails. This is the two-generals problem, and it has no solution — only mitigation.

And this scheduler has more ways to duplicate than a simple ack failure:

• Lesson 7's execution cap terminates a task that may have completed its side effect just before being killed.
• Lesson 4's queue manager makes a task visible again on failure — and "failure" includes a worker that finished but died before deleting the message.
• Lesson 4's cluster manager reassigns tasks from a node it believes failed, when that node may still be running them.
• A network partition can make a healthy worker look dead while it continues working.

Every one of those produces a duplicate execution with no acknowledgment involved at all. So idempotency is not a backstop for unreliable acks — it is the precondition for having a retry mechanism, and this design has four independent sources of retry.

Interview signal by level

Next: running other people's code safely.