Concept Drills: 22 Probes an Interviewer Will Actually Ask

The walkthrough showed these concepts inside one design. This section drills them directly — the follow-up questions interviewers use to find where your understanding stops.

Abstractions and the fallacies

1. RPC hides retries and message packing, yet real systems still define timeout and failure policy at the application level. Why can't those be fully abstracted away? · L5 · Testing: end-to-end argument

Because the framework can resend bytes but cannot know what the call means. Whether a retry is safe depends on business semantics — get_user is harmless to repeat, charge_card is not. Only the application knows that, so the correctness guarantee has to be re-established at the endpoint. Same reason TCP guarantees delivery of bytes but not that the request was processed exactly once: the lower layer can make the guarantee convenient, never sufficient.

2. Name a distributed abstraction you've used and exactly where it leaks. · L5 · Testing: operational depth

A managed queue. It hides leader election, log storage, and replication — genuinely valuable. It leaks in delivery semantics: it's at-least-once, so my consumers must be idempotent. It leaks in ordering: only per-partition, so my partition key is my ordering guarantee. And it leaks in consumer lag: the abstraction says "messages arrive," but I still have to monitor depth and age, and handle poison messages with a dead-letter queue.

3. Which fallacy of distributed computing causes the most production incidents, in your experience? · Staff · Testing: judgment, not recall

"The network is reliable," but the expensive version is "latency is zero" — specifically forgetting that latency is a distribution, not a number. Systems are designed against the average and killed by the tail. Once you fan out to 100 services with a p99 of 100 ms, roughly 63% of requests hit at least one slow path, so the p99 becomes the median. That is a design-time error you cannot fix with a bigger instance.

4. A teammate proposes adding one more synchronous service call to the checkout path. What do you say? · L5 · Testing: availability math

I ask two things: does checkout need that answer to respond, and what's that service's availability? If it isn't needed synchronously, it belongs on a queue. If it is, we've just multiplied its availability into ours — four dependencies at 99.9% each gives 99.6%, about 35 hours a year of inherited downtime. If we take the dependency it needs a tight deadline, a fallback, and a circuit breaker so its bad day isn't automatically our bad day.

RPC and communication

5. When would you choose REST over gRPC? · L4 · Testing: fit, not fashion

At the edge. Browsers can't speak gRPC natively, partners expect HTTP and JSON, and REST responses are cacheable by standard infrastructure and trivially debuggable. Internally, where I control both ends and make billions of calls, gRPC's binary encoding, typed contracts, multiplexing, and streaming are worth the loss of curl-ability.

6. What does an IDL buy you that a JSON API doesn't? · L5 · Testing: contract thinking

A compile-time contract and safe evolution. Both sides generate stubs from one schema, so a field-name typo is a build failure instead of a 3 a.m. null. And because protobuf puts field numbers on the wire rather than names, old readers skip unknown fields — which is what lets client and server deploy independently. The rule that follows: add fields, never renumber or retype them.

7. Your service makes 50 sequential calls to render a page. What do you do? · L4 · Testing: chattiness

Stop making them sequential. Fan out in parallel so I pay the slowest call instead of the sum; better, replace them with one batch call so I pay one round trip. If they're cross-region, neither fixes it — that's 50 × 150 ms of pure network, and the answer is to move the data closer or cache it locally.

8. When is asynchronous messaging the wrong choice? · L5 · Testing: knowing the cost

When the caller needs the answer to respond, when you need a clean synchronous error the user can act on, or when the team won't build the supporting machinery. Async doesn't delete complexity, it relocates it — you now own duplicates, out-of-order arrival, poison messages, dead-letter queues, and lag monitoring. Teams that adopt events to "simplify" and skip that end up with silent data loss instead of loud errors.

Delivery semantics

9. Is exactly-once delivery possible? · L5 · Testing: precision

Not delivery — that's provably impossible over an unreliable network. Exactly-once processing is achievable: deliver at-least-once and deduplicate at the receiver. When a vendor advertises exactly-once, that's what they mean.

10. Design idempotency for a payment endpoint. · Staff · Testing: correctness under concurrency

The client generates a unique key per logical operation and sends it on every attempt including retries — server-generated defeats the purpose. The server claims the key and performs the charge in the same transaction, usually via a unique constraint, so two concurrent retries can't both see "new." And it stores the response, not just a flag, so the retry returns the same answer the first call would have. Miss any of those three and you have a double-charge bug that only appears under retry.

11. Timeout or deadline — what's the difference and which do you want? · L5 · Testing: distributed thinking

A timeout is relative and restarts at every hop, so total latency grows with call depth and is unbounded. A deadline is an absolute instant propagated down the chain, so the whole call tree is bounded and cancels together. Deadlines, always. Without them a client that gave up at 200 ms leaves five downstream services burning capacity on an answer nobody will read.

12. Your dependency slows down and everything falls over. Why, and what prevents it? · Staff · Testing: cascading failure

Callers block on the slow dependency, threads pile up waiting, the connection pool exhausts, and the caller dies too — propagating upstream one service at a time. Retries make it worse by multiplying load on something already struggling. Prevention: bounded concurrency so we shed instead of queueing, a circuit breaker so we fail in microseconds instead of holding a thread for 30 seconds, backoff with jitter, a retry budget capped at a fraction of live traffic, and retries at exactly one layer — three retries at three layers is 27 requests for one logical call.

Consistency

13. Difference between ACID consistency and CAP consistency? · L5 · Testing: precision

Unrelated concepts sharing a word. ACID's C is about the database's own invariants — uniqueness, foreign keys, checks — holding across a transaction on one logical database. CAP's C is about agreement across replicas: every replica showing the same logical value. A system can fully satisfy one and violate the other.

14. Why would a comment system prefer causal consistency over eventual? · L4 · Testing: mapping model to UX

Because eventual consistency puts no ordering on anything, so a reply can become visible before the comment it replies to — users see an answer to a question that doesn't exist yet. Causal consistency preserves exactly the dependent ordering: the reply was written after reading the parent, so that relationship is enforced for every observer. Unrelated top-level comments arriving in different orders for different users is harmless, and causal doesn't pay to order those.

15. Linearizable and serializable — same thing? · Staff · Testing: the classic confusion

No. Linearizability is a recency guarantee about single operations on single objects: reads see the latest write, in real-time order. Serializability is an isolation guarantee about multi-object transactions: the result matches some serial order, not necessarily the real-time one — so serializable alone still permits stale reads. Needing both is strict serializability, which is what Spanner provides and what it pays TrueTime commit-wait for.

16. When is eventual consistency the right answer? · L4 · Testing: not over-engineering

When the cost of a stale read is low and availability matters more. Follower counts, view counts, feeds, DNS, product catalogs. DNS is the largest example running — updates take time to propagate globally, and that's the deliberate price of it being effectively always up.

17. A user updates their profile and doesn't see the change. Diagnose and fix. · L5 · Testing: session guarantees

Their read went to a lagging replica or a stale cache — a read-your-writes violation. The fix is a session guarantee, not strong consistency: the write returns a monotonic version, the client echoes it on reads, and any replica below that version waits or forwards. Key it to the user rather than the connection so it survives a device switch. Reaching for linearizability here would pay quorum latency on every read to fix a small class of requests.

18. "It's distributed, so we're AP." React. · Staff · Testing: whether CAP is understood or recited

Two problems. First, CAP only describes behavior during a partition — it says nothing about the 99.9% of the time the network is fine, where PACELC's "else" applies and you're trading latency against consistency on every request. Second, AP isn't an architecture-wide property. Real systems are CP for money and identity and AP for feeds and carts, often inside one request. I'd want to know which data we're talking about before answering.

Failure

19. Someone in a design review suggests treating late responses as simple crash failures. What's wrong with that? · Staff · Testing: the temporal-vs-crash distinction

Three things break. A crashed node holds no locks and mutates nothing; a slow node holds both and may commit after you declared it dead and elected a replacement — that's split brain. Crash logic assumes departure is terminal, but a node paused by a long GC comes back believing it's still the leader and issues stale writes. And when the cause is load, evicting it pushes its traffic onto already-struggling peers and turns one slow node into a cluster outage. The right response to slow is shed, hedge, and fence — attach a monotonic epoch token to leadership so storage rejects the zombie's writes.

20. Your service is green on every dashboard and customers say it's broken. What's happening? · Staff · Testing: gray failure

Gray failure — the system's view of its own health disagrees with the users' view. Health checks confirm the process is up and the port is open; they say nothing about a 4% timeout rate or a p99 of 8 seconds. Shallow checks only detect crash failures, and are blind to omission and temporal ones, which are the failures that hurt. The fix is measuring what the client experiences — real request success rate and latency from the caller's side — and alerting on that rather than on server self-reports.

21. How many replicas to tolerate f failures, and does the answer change for Byzantine faults? · Staff · Testing: quorum reasoning

2f + 1 for crash faults — a majority survives and any two quorums intersect. 3f + 1 for Byzantine, because honest nodes must both outvote the liars and still form a quorum without them. Inside one company's datacenter I assume crash faults and defend the realistic Byzantine subset with checksums for bit rot and schema validation for bad deploys. Full BFT is for genuinely adversarial participants — blockchains, aerospace, cross-organization consensus.

22. A node passes health checks but drops 30% of real requests. Which failure model, and how do you catch it? · L5 · Testing: omission failures

Omission failure — partly alive, which is worse than dead because it keeps its place in the load balancer pool. A shallow health check will never catch it. You catch it with per-node success-rate monitoring rather than aggregate metrics, which will hide one bad node in a large fleet, and with outlier detection at the load balancer that ejects a node whose error rate diverges from its peers.

Self-check

If you can answer these cold, you have the foundations:

The cheat sheet next compresses all of it into one page for the morning of an interview.