Idempotency and the Timeout Ambiguity

In one line: this is the strongest section of the design, and it is the single most important concept in payment system design. It is also incomplete in a way that matters.

The ambiguity

If a customer's payment request times out, it's unclear what happened on the backend:

1. The payment was processed successfully, but the response was lost on its way back to the client.
2. The request is still being processed, but the client-side timeout was reached.
3. The request was lost in transit and never reached the payment service.

The three cases are indistinguishable from the client, and they need opposite responses

This is the crux, and the design states it precisely: "the client cannot know the true status of the payment."

From the client's position, all three look identical — a request sent, no response, a timer expired. But the correct action differs completely:

Case 1 (succeeded, response lost):  DO NOT retry -> retrying double-charges
Case 3 (never arrived):             DO retry     -> not retrying loses the sale
Case 2 (still in flight):           Retrying may double-charge, may not,
                                    depending on a race you cannot observe

You must choose an action, the information needed to choose is unavailable, and one wrong choice charges someone twice.

Notice this is not a payments-specific problem — it is a property of any unreliable network. You cannot distinguish "the request was lost" from "the response was lost" without additional state. The reason it dominates this chapter and not the others is the asymmetry of the cost: that building block's duplicate build wasted CPU; here it takes money from a real person.

The insight that resolves it: stop trying to determine what happened, and make it not matter. That is what idempotency does.

Idempotency keys

An idempotent API ensures that repeating the same request multiple times produces the same result as the initial request, preventing duplicate charges. This is typically implemented by having the client generate a unique request ID (an idempotency key) that the server uses to recognize and de-duplicate retried requests.

Client-generated is the load-bearing word

The design gets this exactly right, and it is the detail most people get wrong.

The key must be created by the client, before the first attempt, and reused unchanged for every retry of that same logical operation:

Attempt 1:  POST /payments  Idempotency-Key: 7f3a-9c21-...   -> timeout
Attempt 2:  POST /payments  Idempotency-Key: 7f3a-9c21-...   -> SAME key
Attempt 3:  POST /payments  Idempotency-Key: 7f3a-9c21-...   -> SAME key

The server can now recognize attempts 2 and 3 as the same operation and return the original outcome instead of charging again.

Server-generated keys cannot work, because the server assigning an ID on arrival gives each retry a different one — every attempt looks new, which is precisely the failure being prevented.

Two properties follow:

The key must be stable across retries and unique across operations. A UUID minted when the user opens the checkout page satisfies both. A hash of the request body does not — a customer legitimately buying the same item twice would be silently deduplicated into one purchase.

The key ties the retry to an intent, not to a request. That is why the window matters: the same key returns the same result for hours or days, so a retry after a network partition heals still resolves correctly.

Idempotency requires the caller to name the operation before attempting it — which is why it is an API contract rather than a server-side implementation detail.

The design's suggestion of an order number as the key is subtly wrong

"We can create a distinct transaction identifier (like an order number or payment ID) for every transaction, which serves as our idempotence key."

The mechanism described afterwards is right — store the key with a status of succeeded, failed, or in progress, and reject duplicates. The choice of key is where it slips.

An order number identifies the order, not the payment attempt. They are not one-to-one:

Order #1234, attempt 1: card declined (insufficient funds)
Order #1234, attempt 2: customer tries a different card
  -> SAME order number, but a genuinely DIFFERENT operation
  -> keyed on the order number, attempt 2 is rejected as a duplicate
     and the customer cannot pay at all

The key must scope to one attempt-set: one intent to charge, retried until it resolves. A new payment attempt after a decline is a new intent and needs a new key.

Getting this wrong produces the mirror-image failure of a double charge — a customer who cannot complete a legitimate purchase — which is less alarming and more common.

Scope the idempotency key to the retryable attempt, not to the business object it belongs to.

How it works on the server

The unique constraint is the mechanism — an application-level check is not enough

The naive implementation is "look up the key; if absent, process." That has a race:

Retry A: SELECT key -> not found
Retry B: SELECT key -> not found        <- both passed the check
Retry A: process -> CHARGE
Retry B: process -> CHARGE AGAIN

Two concurrent retries — routine when a client times out and immediately retries against a different server behind a load balancer — both see an empty result and both proceed.

The fix is to make the database enforce it: a UNIQUE constraint on the idempotency key, and an insert-first protocol. Exactly one insert can succeed; the loser learns it is a duplicate from the constraint violation, not from a read.

That is why Lesson 5's ledger schema puts idempotency_key UNIQUE on the entry table itself. Enforcing uniqueness at the same place the write lands is the only version that has no race.

The in progress state is the third case from the timeout ambiguity, and it needs its own answer: the original is still running, so returning "success" would be a lie and processing again would double-charge. Return a retry-after and let the client ask again — which is exactly what checkPaymentStatus(transaction_id) is for.

Idempotency is a uniqueness constraint plus a stored result, and the constraint must live in the database.

What the chapter leaves out

Idempotency is described only for the client, and it is needed between every pair of services

The design treats this as a client-facing concern — the customer clicks buy twice, or the browser retries. That is the visible case and it is not the dangerous one.

Every hop in the chain has the same ambiguity, and most of them retry automatically:

Client        -> Payment service      <- the design covers this one
Payment svc   -> Payment gateway      <- times out, retries: DOUBLE AUTHORIZATION
Gateway       -> Card network         <- external, opaque, retries
Kafka         -> Wallet consumer      <- at-least-once: DOUBLE CREDIT (Lesson 7)
Payment svc   -> Ledger               <- DOUBLE ENTRY

The payment service calling the gateway faces exactly the three cases in the diagram above, with the same inability to distinguish them — and it will hit them far more often than a customer double-clicks, because it is making thousands of calls a second across a network it does not own.

Which is why real payment APIs — Stripe, Adyen, Braintree — require an idempotency key on every mutating request. Not offer it: require it.

And this is the connection back to the previous chapter. that building block's heartbeat lease could reassign a job to a second worker, and the fix was a fencing token so the stale worker's write was rejected. Same structure, different mechanism:

Fencing token:    reject the write from a stale actor
Idempotency key:  recognize the write as one already performed

Both accept that you cannot prevent duplicate execution and must instead make duplicate effects impossible. In a build system "probably equivalent artifacts" was acceptable. Here it is not, so the guarantee has to be exact.

Idempotency belongs on every mutating interface in a payment system, not only the one facing the customer.

checkPaymentStatus is the other half of the answer, and the design doesn't wire it up

Idempotency makes retrying safe. It does not tell the client what happened — and sometimes the client needs to know rather than retry.

checkPaymentStatus(transaction_id) from Lesson 3 is exactly that mechanism: after a timeout, instead of retrying, ask. Combined with the state machine in Lesson 3, a client can resolve all three cases without risk:

timeout -> checkPaymentStatus(id)
  status Authorized/Captured -> case 1: it worked. Show success.
  status Initiated           -> case 2: still running. Wait and poll.
  not found                  -> case 3: never arrived. Safe to retry.

That is a complete resolution of the ambiguity, using two mechanisms the design already specifies — and the design never connects them. Nothing in the workflow describes a client that times out, queries status, and acts on the answer.

Idempotency makes retries safe; a status endpoint makes them unnecessary. Production systems provide both, and the second is what turns a timeout from a guess into a lookup.

On choosing the timeout value

The design's advice — base it on how long payments typically take, follow the gateway's guidance, balance user experience against risk — is reasonable and vague, and one clause is off: "if it's too large, it could be a security risk." A long timeout is a resource problem (connections and threads held open, per Lesson 2's concurrency estimate), not a security one.

The sharper framing: the timeout is a bet about which failure you prefer.

TOO SHORT: you abandon requests that would have succeeded
           -> more ambiguity, more retries, more load on a system already slow
TOO LONG:  you hold resources for calls that will never return
           -> connection pool exhaustion, cascading failure

And the rule that matters most: your timeout must be shorter than your caller's, all the way down the chain. If the payment service waits 30 seconds on the gateway while the browser gives up at 10, the client is retrying into work that is still running — manufacturing case 2, the worst of the three.

Timeout budgets must decrease as you go down the call stack, and this design's three external hops make that a real constraint rather than a nicety.

Next: at-least-once delivery meeting a mutable balance.