Transient Failures — and a Fallback That Fails Open

In one line: with three opaque external hops, failure handling is the design. Two of the chapter's three strategies are right, and the third is exploitable.

Retry

The retry strategy helps overcome temporary network issues. We can implement exponential backoff and set a maximum number of retries. If the transaction still fails after reaching the limit, it is marked as failed.

Exponential backoff is right, and it needs two additions the design omits

Backoff with a retry cap is the correct baseline. A fixed-interval retry against a struggling dependency is how you convert a brown-out into an outage — every client hammering at a constant rate keeps the service saturated exactly when it is trying to recover.

Two things are missing.

Jitter. Without it, all the clients that failed at the same moment retry at the same moment:

NO JITTER:  1,000 clients fail at t=0
            all retry at t=1s, all at t=2s, all at t=4s
            -> synchronized thundering herd on every round

JITTER:     each waits a RANDOM interval within the backoff window
            -> retries spread out; the dependency sees a smooth curve

A backoff schedule without jitter reproduces the spike it was designed to prevent. It is one line of code and it is the difference between recovery and a retry storm.

A circuit breaker. Backoff paces an individual client; it does not stop the system from retrying into a dependency that is comprehensively down. After a threshold of consecutive failures, stop calling entirely for an interval, fail fast, and periodically probe with a single request to test recovery.

This matters enormously here because of Lesson 2's concurrency figure: 1,400 TPS × 500 ms means roughly 700 concurrent outbound calls in flight. If the gateway slows from 500 ms to 5 seconds, in-flight calls climb tenfold, the connection pool exhausts, and the payment service fails for everyone — including the requests that would have succeeded. A circuit breaker is what bounds that blast radius.

Backoff protects the dependency; a circuit breaker protects you from the dependency. You need both.

Retrying a payment is only safe if the retried call is idempotent — and that is not stated here

The retry section and the idempotency section are separate parts of the chapter, and the connection between them is never made.

Retrying an authorization means sending the same charge again to the gateway. Per Lesson 6, the failure that triggered the retry is ambiguous — the first attempt may have succeeded with a lost response. So a retry without an idempotency key is a mechanism for double-charging, deployed deliberately and at scale.

Retry WITHOUT an idempotency key -> exponential backoff toward duplicate charges
Retry WITH    an idempotency key -> safe, and the retry limit is just a give-up point

This is the same pairing as the previous chapter, where retrying a build was safe because builds are near-idempotent and the gap was only a fencing token. Here nothing is naturally idempotent, so the key is mandatory.

A retry policy without an idempotency guarantee is not a resilience strategy, it is an amplifier. Say them together, always.

And one more: not everything should be retried. A 402 insufficient funds or a declined from the issuer is a terminal answer — retrying it wastes calls, may trip the issuer's velocity checks, and can look like card testing. Retry timeouts, connection failures, and 5xx; never retry a business decline.

Timeout

The three-way ambiguity is covered fully in Lesson 6. The one point to add here is the budget relationship:

Timeout budgets must shrink as you go down the call stack

With three external hops the constraint is real rather than theoretical:

Browser        -> Payment service:   10 s
Payment svc    -> Gateway:            8 s      <- must be LESS
Gateway        -> Card network:       5 s      <- must be LESS

Invert any of those and the caller gives up while the callee is still working — which manufactures case 2 from the timeout ambiguity, the worst of the three, because the operation is genuinely in flight and its outcome is genuinely unknown.

The design's own advice — "if it's too large, it could be a security risk" — misidentifies the cost. A long timeout is a resource problem: connections, threads, and memory held open for calls that will never return, which is how connection-pool exhaustion cascades.

A timeout is a bet about which failure you prefer: too short abandons requests that would have succeeded and adds load to a system already slow; too long holds resources for calls that are already dead.

Fallback

A fallback is a backup process used when a primary method fails. For example, if the fraud check service returns an error, instead of failing the entire payment, we can use a fallback. For a very small transaction, the fallback might be to approve the payment and accept the minor risk. This approach balances risk against customer experience.

This is fail-open on fraud detection, and it converts an outage into an attack

The general principle — degrade gracefully rather than failing entirely — is sound. This specific application of it is not, and it is the most consequential defect in the chapter.

A fraud check that approves when it errors is a fraud check an attacker can disable.

1. Attacker degrades or overloads the fraud detection service
2. Fraud checks now return errors
3. Fallback approves everything
   -> the control is off, and the system reports success

You have made availability of the security control optional, which means the attacker's cheapest path is no longer to evade detection but to remove it.

And the qualifier makes it worse rather than better. "For a very small transaction" describes card testing precisely: attackers validate stolen card numbers with tiny charges — often under a dollar — to find which cards are live before using them for real purchases. Small transactions are not the low-risk case for fraud; they are the signature of the highest-volume fraud pattern there is.

Security controls fail closed. If you cannot determine whether a transaction is fraudulent, you do not assume it is fine.

The graceful degradation that is available here uses the risk score from Lesson 4:

Fraud service unavailable ->
  - fall back to a CACHED risk score for a known customer, if recent
  - apply a STRICTER static policy: known device + established history + low velocity
  - queue for asynchronous review and delay CAPTURE rather than declining outright
  - decline, and tell the customer to try again shortly

The third option is the elegant one and it is available only because payments have two phases: authorize now, hold the capture until the fraud service recovers and scores it. The customer's purchase succeeds, the money does not move, and the risk is bounded by the hold rather than by a guess. That is the two-phase model from Lesson 1 paying for itself — and it is unavailable to this design, which never implements capture.

Fail closed on anything that decides whether an action is permitted; fail open only on things that decide how nicely it is presented.

Where fallback IS the right answer

The technique is not wrong, only misapplied. Fallbacks belong where degradation is genuinely benign:

The dividing line is exactly the one above: components that decide whether something is allowed must fail closed; components that decide how good the experience is may fail open.

Note the receipt row, because it is the shape you want more often than a fallback: if a dependency is not required for correctness, take it off the critical path entirely. Then its failure is not something to fall back from — it is a queue that drains later.

The three strategies together

Persistent failures need a different answer from transient ones

The design poses "how can we avoid persistent failures?" as its third reliability question and never returns to it.

The distinction matters because the strategies above only help with transient failures. A dependency that is comprehensively down does not recover because you retried it more politely.

What persistent failure needs:

Circuit breakers, so you stop spending resources on calls that will fail.

A dead-letter queue. Events that fail after all retries must go somewhere durable and inspectable — not be dropped, and not be retried forever. For payments this is mandatory: an event that cannot be processed represents money in an unknown state, and a human has to look at it.

Multi-provider failover. The most robust answer for a payment system, and the one most operators eventually build: integrate more than one PSP and route around a failed one. It is expensive — two integrations, two reconciliation pipelines, two sets of settlement files — and it is what removes the single largest external dependency from the critical path.

Reconciliation as the backstop, which the design does have: whatever the failure handling misses, the daily comparison against the settlement file surfaces.

Transient failures are handled by retrying; persistent failures are handled by routing elsewhere and by making the unresolved cases visible.

Next: reconciliation and disputes.