RPC Failure Semantics: At-Most-Once, At-Least-Once, Exactly-Once

Why this matters: the RPC runtime retransmits on your behalf. If the call was charge_card, that helpful retransmission just billed your customer twice. This lesson is the single largest gap between an engineer who uses RPC and one who can be trusted to design with it.

Key takeaway

When an RPC times out, you do not know whether it ran. Every remote call therefore has a delivery semantic — at-most-once, at-least-once, or "exactly-once" — and you must choose it deliberately per operation. The default in every retrying framework is at-least-once, which is only safe if the operation is idempotent.

The three-outcome problem

A local call has two outcomes. A remote call has three.

The client cannot distinguish these three worlds:

The request never reached the server.
The request ran, and the response was lost.
The request is still running, slowly.

From the client's side all three look identical: silence. That ambiguity is not a bug in your RPC library — it is a theorem about networks. No amount of framework engineering removes it.

The three delivery semantics

Semantic	How it's implemented	Risk you accept	Use it for
At-most-once	Send once. Never retry.	The operation may silently not happen	Non-critical, high-volume, replaceable work — metrics, telemetry, cache warming
At-least-once	Retry until acknowledged	The operation may run more than once	Almost everything — safe ONLY if the handler is idempotent
Exactly-once (effective)	At-least-once delivery + server-side deduplication	Complexity and dedup-state storage	Payments, order placement, inventory decrement

Idempotency: the property that makes retries safe

An operation is idempotent if performing it N times has the same effect as performing it once.

Operation	Idempotent?	Why
GET /users/42		Reads change nothing
PUT /users/42 {name: 'Ana'}		Sets an absolute value — same result every time
DELETE /users/42		Second delete is a no-op (return success, not 404)
balance = balance - 50		Relative change — compounds on every retry
POST /orders		Creates a new resource per call — unless you add a key

The general fix is an idempotency key: the client generates a unique ID for the logical operation and sends it with every attempt, including retries. The server records which keys it has completed.

Three details that decide whether this actually works in production:

The client generates the key, not the server. A server-generated key would be new on each retry, defeating the purpose.
Claim the key and do the work atomically — usually a unique constraint in the same transaction as the effect. Otherwise two concurrent retries both see "new" and both charge.
Store the response, not just a flag. The retry must return the same answer the first call would have, or the client sees an error for work that succeeded.

Timeouts and deadlines

A retry policy is meaningless without a timeout, and there are two ways to express one.

Approach	What the caller says	Behavior across hops	Verdict
Timeout	"Give up after 200 ms"	Each hop restarts its own 200 ms clock	Total time grows with call depth — unbounded
Deadline	"Give up at 10:00:00.200"	Absolute instant propagates down the whole chain	Bounded end to end — prefer this

Deadline propagation is the mechanism: service A gets a 300 ms deadline, spends 80 ms, and calls B with the remaining 220 ms. B calls C with what is left after its own work. When the deadline passes, every service in the chain abandons the request simultaneously.

Retry storms and how to survive them

Retries are the correct response to a transient failure and the worst response to an overloaded service. When a dependency slows down, every caller times out and retries — tripling load on a service that was already failing.

Backoff with full jitter:
  delay = random(0, min(cap, base * 2^attempt))

The 2^attempt gives exponential backoff, which drains pressure. The random(...) gives jitter, which is the part engineers skip and must not: without it every client retries at the same instant, and you have rebuilt the thundering herd with extra steps.

Three more controls, in increasing order of maturity:

Retry budget — cap retries at a fraction of live traffic (say 10%). Under a broad outage, retries stop amplifying instead of scaling with the failure.
Retry only at one layer. Retries at the client, the gateway, the mesh, and the library multiply: three retries at three layers is 27 requests for one logical call. Pick a layer; disable the rest.
Circuit breaker — stop calling a service you have good reason to believe is down.

Closed passes traffic. Once the failure rate crosses a threshold the breaker trips Open and calls fail instantly — which protects the caller's threads and gives the dying dependency room to recover. After a cool-down it goes Half-Open and allows a single trial request to decide whether to close or re-open.

Putting the policy together

For any remote call, decide these five things explicitly:

Decision	Question to answer	Common default
Semantic	Is duplicate execution acceptable?	At-least-once + idempotency key
Deadline	When is the answer worthless?	Propagated absolute deadline
Retries	Is this failure transient?	2 retries, exponential backoff, full jitter
Breaker	What if it's down, not slow?	Trip at ~50% error rate over a rolling window
Fallback	What do we serve when it fails?	Stale cache, degraded response, or a clean error