RPC Failure Semantics: At-Most-Once, At-Least-Once, Exactly-Once
In one line: the RPC runtime retransmits on your behalf. If the call was charge_card, that helpful retransmission just billed your customer twice. This lesson is the single largest gap between an engineer who uses RPC and one who can be trusted to design with it.
The three-outcome problem
A local call has two outcomes. A remote call has three.
The client cannot distinguish these three worlds:
- The request never reached the server.
- The request ran, and the response was lost.
- The request is still running, slowly.
From the client's side all three look identical: silence. That ambiguity is not a bug in your RPC library — it is a theorem about networks. No amount of framework engineering removes it.
The three delivery semantics
| Semantic | How it's implemented | Risk you accept | Use it for |
|---|---|---|---|
| At-most-once | Send once. Never retry. | The operation may silently not happen | Non-critical, high-volume, replaceable work — metrics, telemetry, cache warming |
| At-least-once | Retry until acknowledged | The operation may run more than once | Almost everything — safe ONLY if the handler is idempotent |
| Exactly-once (effective) | At-least-once delivery + server-side deduplication | Complexity and dedup-state storage | Payments, order placement, inventory decrement |
Idempotency: the property that makes retries safe
An operation is idempotent if performing it N times has the same effect as performing it once.
| Operation | Idempotent? | Why |
|---|---|---|
| GET /users/42 | Reads change nothing | |
| PUT /users/42 {name: 'Ana'} | Sets an absolute value — same result every time | |
| DELETE /users/42 | Second delete is a no-op (return success, not 404) | |
| balance = balance - 50 | Relative change — compounds on every retry | |
| POST /orders | Creates a new resource per call — unless you add a key |
The general fix is an idempotency key: the client generates a unique ID for the logical operation and sends it with every attempt, including retries. The server records which keys it has completed.
Three details that decide whether this actually works in production:
- The client generates the key, not the server. A server-generated key would be new on each retry, defeating the purpose.
- Claim the key and do the work atomically — usually a unique constraint in the same transaction as the effect. Otherwise two concurrent retries both see "new" and both charge.
- Store the response, not just a flag. The retry must return the same answer the first call would have, or the client sees an error for work that succeeded.
Timeouts and deadlines
A retry policy is meaningless without a timeout, and there are two ways to express one.
| Approach | What the caller says | Behavior across hops | Verdict |
|---|---|---|---|
| Timeout | "Give up after 200 ms" | Each hop restarts its own 200 ms clock | Total time grows with call depth — unbounded |
| Deadline | "Give up at 10:00:00.200" | Absolute instant propagates down the whole chain | Bounded end to end — prefer this |
Deadline propagation is the mechanism: service A gets a 300 ms deadline, spends 80 ms, and calls B with the remaining 220 ms. B calls C with what is left after its own work. When the deadline passes, every service in the chain abandons the request simultaneously.
The work nobody is waiting for
Without deadline propagation, a client that gave up at 200 ms leaves five downstream services still grinding on a request whose answer will be thrown away. Under load, a large fraction of a struggling fleet's capacity can be consumed by work no one will ever read. Propagated deadlines let the whole chain cancel together.
Retry storms and how to survive them
Retries are the correct response to a transient failure and the worst response to an overloaded service. When a dependency slows down, every caller times out and retries — tripling load on a service that was already failing.
Backoff with full jitter: delay = random(0, min(cap, base * 2^attempt))
The 2^attempt gives exponential backoff, which drains pressure. The random(...) gives jitter, which is the part engineers skip and must not: without it every client retries at the same instant, and you have rebuilt the thundering herd with extra steps.
Three more controls, in increasing order of maturity:
- Retry budget — cap retries at a fraction of live traffic (say 10%). Under a broad outage, retries stop amplifying instead of scaling with the failure.
- Retry only at one layer. Retries at the client, the gateway, the mesh, and the library multiply: three retries at three layers is 27 requests for one logical call. Pick a layer; disable the rest.
- Circuit breaker — stop calling a service you have good reason to believe is down.
Closed passes traffic. Once the failure rate crosses a threshold the breaker trips Open and calls fail instantly — which protects the caller's threads and gives the dying dependency room to recover. After a cool-down it goes Half-Open and allows a single trial request to decide whether to close or re-open.
Failing fast is a feature
An open circuit returns an error in microseconds instead of tying up a thread for a 30-second timeout. This is what prevents a cascading failure: without it, threads in the caller pile up waiting on the dead dependency until the caller exhausts its pool and dies too — propagating the outage upstream, one service at a time.
Putting the policy together
For any remote call, decide these five things explicitly:
| Decision | Question to answer | Common default |
|---|---|---|
| Semantic | Is duplicate execution acceptable? | At-least-once + idempotency key |
| Deadline | When is the answer worthless? | Propagated absolute deadline |
| Retries | Is this failure transient? | 2 retries, exponential backoff, full jitter |
| Breaker | What if it's down, not slow? | Trip at ~50% error rate over a rolling window |
| Fallback | What do we serve when it fails? | Stale cache, degraded response, or a clean error |
Key takeaway
Retry only what is safe to retry, only for as long as the answer is useful, only at one layer, with jitter, and stop entirely when the dependency is clearly down. That sentence is most of production reliability engineering. And when two services must both succeed, 2PC is correct and usually the wrong tool — participants block holding locks between the vote and the decision, so the coordinator becomes a single point of failure with everyone's locks. Sagas give up atomicity for compensating actions that never block, at the cost of a visibly inconsistent window. Distributed atomicity is available and you usually should not buy it.
When one call is not enough: writing to two places
Everything above concerns a single call that may or may not have happened. The harder version is two calls that must both happen or neither — charge the card and reserve the seat, debit one account and credit another — where the services own separate data and there is no shared transaction to wrap them in.
Ordering does not save you; it only chooses which inconsistency you get. There are two real answers.
Two-phase commit: correct, and usually the wrong tool
A coordinator asks every participant to prepare — do the work, hold the locks, promise you can commit — and only when all say yes does it tell them to commit.
It is genuinely correct, and the failure mode is why it is rare across service boundaries. Between voting and hearing the decision, a participant is blocked holding locks and cannot resolve on its own: it does not know whether the others voted yes. A coordinator that dies in that window leaves every participant stuck, and availability is now bounded by the coordinator's — a single point of failure with everyone's locks in its hand.
It also requires every participant to support the protocol, which rules it out the moment one of them is a third-party API.
Sagas: give up atomicity, keep the outcome
The alternative accepts that the steps commit independently and adds a way to walk them back.
Each step has a compensating action that semantically undoes it, and on failure the saga runs them in reverse. Nothing blocks, nothing holds a lock across services, and no participant needs to speak a special protocol.
What you give up is isolation, and it is worth being blunt about it: there is a window where the seat is reserved and the payment is refunded, and anyone looking during that window sees a state that never should have existed. The system is eventually consistent, not atomic.
Three practical consequences to volunteer:
A compensation is not a rollback. You cannot un-send an email or un-charge a card — you send a correction, or issue a refund. The compensating action is a new business event with its own record, which is often what auditors actually want.
Compensations can fail too, so they need retries and, ultimately, a human. A saga stuck half-compensated is the worst state, which is why the orchestration is usually durable — a workflow engine or a state machine in a database, not in-memory in whichever service happened to start it.
Every step must be idempotent, because retries apply here exactly as they do to a single call. The saga is built out of the at-least-once machinery from earlier in this lesson.
| Two-phase commit | Saga | |
|---|---|---|
| Guarantee | Atomic — all or nothing, never visibly partial | Eventually consistent — visibly partial in between |
| Locks | Held across services until the decision | None held across services |
| Coordinator failure | Participants block, holding locks | Resume the workflow; nothing is stuck |
| Third-party participants | Impossible — they must implement the protocol | Fine, if the action can be compensated |
| Reach for it when | Inside one database or one tightly-coupled cluster | Across services, which is nearly always |
The judgement worth carrying: distributed atomicity is available and you usually should not buy it. The price is blocking on a coordinator, and most business processes tolerate a brief inconsistency far better than they tolerate an outage.
Interview signal by level
| Level | What a strong answer sounds like |
|---|---|
| L4 | "We retry failed calls with a timeout." |
| L5 | "Exponential backoff with jitter, and the payment endpoint takes an idempotency key so retries don't double-charge." |
| Staff+ | "Exactly-once delivery isn't achievable, so: at-least-once plus server-side dedup, the key claimed in the same transaction as the effect, propagated deadlines so the chain cancels together, retries budgeted at one layer, and a breaker so we fail fast instead of cascading." |
Next: when not to use RPC at all.