The Spectrum of Failure Models

Why this matters: you cannot design fault tolerance without first saying precisely what a fault is. "The server might go down" is not a specification. Failure models are the specification — and each one demands a different mitigation.

Key takeaway

Failures in distributed systems are inevitable and may be transient or permanent. Failure models give you a framework for reasoning about a failure's impact and choosing the right mitigation. The models get strictly harder to handle as you move right along the spectrum.

The spectrum

Difficulty increases left to right, and the reason is consistent: detectability. A failure you can reliably detect is a failure you can route around. Everything on the right side is hard precisely because the healthy nodes cannot tell what is happening.

Model	What the node does	Can others detect it?	Standard mitigation
Fail-stop	Halts permanently, detectably	Yes — reliably	Remove from the pool, fail over
Crash	Halts silently	Not directly — inferred	Heartbeats plus timeouts
Omission	Drops messages it should send or receive	Partially — looks like packet loss	Acknowledgments, retries, sequence numbers
Temporal	Correct results, too late to be useful	Only against a deadline	Deadlines, hedged requests, load shedding
Byzantine	Arbitrary — wrong data, lies, random behavior	Only by cross-checking replicas	Checksums, signatures, BFT quorums

Fail-stop

In a fail-stop failure, a node halts permanently, and other nodes can reliably detect the termination by communicating with it. Because the failure is detectable, fail-stop is the simplest model to handle: the system knows exactly who is gone and can redistribute the work.

This is the model most textbook algorithms assume — and the model real hardware rarely provides. It is closest to reality when a supervisor or orchestrator explicitly kills and reports a process.

Crash

In a crash failure, a node halts silently. Unlike fail-stop, other nodes cannot readily detect that it stopped working.

That single word — silently — is what makes distributed systems hard. From the outside, a crashed node is indistinguishable from a node that is merely slow, or reachable but partitioned away from you. You are forced to infer death from absence of evidence:

Omission failures

Omission failures occur when a node fails to send or receive messages. Two flavors:

Send omission: the node fails to respond to an incoming request.
Receive omission: the node fails to receive a request and therefore cannot acknowledge it.

These are nastier than crashes because the node is partly alive. It passes health checks, holds its place in the cluster, and drops a fraction of real work. A node that answers /healthz while silently dropping 30% of production requests will stay in your load balancer pool indefinitely.

Temporal failures

In a temporal failure, a node produces correct results but delivers them too late to be useful. Causes include inefficient algorithms, poor design choices, and loss of synchronization between processor clocks.

This is the dominant failure mode in modern systems, and the least respected. A node returning correct answers at 30 seconds when the deadline is 200 ms is, functionally, down — while looking perfectly healthy on every dashboard that tracks error rate.

Why you must not treat 'late' as 'crashed'

It is tempting to collapse temporal into crash: no answer in time, so call it dead. That assumption breaks in three specific ways.

The work is still running. A crashed node holds no locks and mutates nothing. A slow node holds both — and may commit its write after you declared it dead and elected a replacement. Two nodes now believe they own the same data: split brain.
The node comes back. Crash-failure logic assumes departure is terminal. A node paused by a 40-second GC returns believing it is still the leader, and starts issuing stale writes with full confidence.
The cause is load, and eviction makes it worse. Declaring an overloaded node dead redistributes its traffic onto its already-struggling peers, which pushes them past their own thresholds. That is how a slow node becomes a cluster-wide outage.

The correct response to slow is shed load, hedge the request, and fence the node — not simply "replace it."

The mitigation for the split-brain half is fencing: attach a monotonically increasing token to leadership, and have storage reject any write carrying a stale token. The returning zombie leader is then harmlessly ignored rather than trusted.

Byzantine failures

In a Byzantine failure, a node exhibits arbitrary behavior. It may transmit random messages, produce incorrect results, or stop unexpectedly. These failures typically stem from malicious attacks or software bugs, and they are the most difficult to mitigate — because the node is not silent, it is actively wrong, and confidently so.

Every other model assumes a failed node stops contributing. Byzantine assumes it contributes garbage that looks legitimate.

Gray failure: the model the spectrum misses

The hardest real-world failures are not any single point on the spectrum — they are the ones where the system's own view disagrees with its users' view.

Observer	What it sees	Conclusion
Health-check monitor	Process up, port open, 200 OK	Healthy
Peer nodes	Heartbeats arriving, a little late	Healthy
Actual users	p99 at 8 seconds, 4% of requests timing out	Broken

This is gray failure, and it produces the outages that last hours — because every automated system reports green while the pager is silent and the customers are not. The defense is measuring what the user experiences (success rate and latency of real requests, from the client's side) rather than what the server believes about itself.

Choosing your model

You do not defend against all five. You declare which you handle and design accordingly:

Nearly every system you build inside one organization lands on crash + temporal: assume nodes stop or get slow, assume they do not lie, checksum anything that crosses a disk or a wire.