Interview Walkthrough: Detect Client-Side Errors

"Users are reporting they can't reach your service, but all your dashboards are green. Design a system to detect this."

This framing is common, and it is a good one — it hands you the symptom and asks for the mechanism. It also appears as a follow-up to any monitoring or availability question.

Step 0 — Scope it before you design

• Do we control the client — a native app, or just a website in someone else's browser?
• What scale of user base, and how geographically distributed?
• Are we detecting any unreachability, or specific classes like DNS and routing?
• What's the detection latency requirement — seconds, minutes?
• What are the privacy and regulatory constraints on collecting from clients?

Then commit:

"I'll assume a large global user base, a mix of web and native clients, and that we want near-real-time detection of users who can't reach us at all. Privacy constraints are strict — I'll design for the browser case and note where owning the client gives us more latitude."

Step 1 — Establish why this is hard

Do this first. It justifies everything after it.

"Server-side failures are easy — elevated HTTP 500s in the web and application logs. Client-side failures are the opposite: if DNS fails, or routing sends traffic somewhere else, or a middlebox drops it, the request never arrives and there is no log line to find. Our infrastructure is genuinely healthy, so every internal signal is correctly green.

Three classes cause it: DNS resolution failures, routing failures between the client and us, and third-party infrastructure like middleboxes or CDNs."

Step 2 — Kill the obvious answer

"The instinct is to alert on a traffic dip. That doesn't work, for two reasons. Traffic varies naturally by time of day and week, so a dip-based threshold either fires constantly or is too loose to catch anything. And more importantly, when the problem affects one region or one ISP, that might be 2% of global traffic — well inside normal variance.

The signal is a small population disappearing from a large one, which is exactly what aggregate metrics hide."

Step 3 — Active probing, and its ceiling

"First real design: act like a client. Deploy probers at global vantage points that periodically request the service, so we can tell 'unreachable from Frankfurt' from 'globally down.' That gets us the external vantage point we need.

But it has a hard ceiling. The internet has over 100,000 autonomous systems, and we can realistically run probes in a few hundred places — cost, regulations, and maintenance make more infeasible. So we sample a tiny fraction of the paths users actually take, and the failures that matter are concentrated in networks we don't have a probe in. It's also synthetic traffic, which says little about a real user on a mobile network behind a corporate proxy.

Probing gives us presence, not coverage. Keep it — it's cheap, needs no consent, and works before we have users somewhere. But it can't be the only mechanism."

Step 4 — Agent and collector

"Invert it. Instead of us sampling the network, embed an agent in the client that reports when a request fails, and stand up a collector to receive those reports.

The collector must be independent of the primary service — that's the load-bearing constraint. If reports went to the main service they'd fail for exactly the same reason as the original request.

This fixes the coverage problem by construction: reports come from every network that has a real user in it, not from wherever we happened to place a prober.

For scale: collectors deployed close to client networks to cut latency, hierarchical aggregation upward, stream processing for near-real-time spike detection, and sampling at around 1% to control cost — which works because we're estimating a rate, not reconstructing individual sessions."

Step 5 — The three consequences

"Putting code on someone else's device raises three things, and two of them are about the user rather than the system.

Consent. A custom HTTP header signals the client supports reporting, sent only if the user already consented. The service responds with the policy and collection endpoint — as data, so we can move collectors without shipping a client. Note this needs browser support to work on the web; in our own native app it's trivial. Whether we control the client changes this materially.

Failure domains. The collector has to survive the failure it's reporting: different IP, different domain, different autonomous system. Those compose, and each one we skip is a failure class we go blind to. For last-mile loss — the user has no connectivity at all — there's no fall-back, so the agent buffers locally and forwards on recovery.

Privacy. The most diagnostic fields are the most identifying: traceroute reveals location, resolver identity leaks location, RTT can fingerprint. So minimize. The test I'd apply: would this field already appear in our weblogs on a successful request? If not, a failure isn't a licence to collect it."

Deep Dives & Follow-up Questions

"How do you distinguish 'no errors' from 'nobody can report errors'?"

That's the ambiguity at the centre of this design, and it's why probing stays in the mix. If agent reports drop to zero, that could mean everything is fine or that the collector itself is unreachable. Probers give an independent signal with a different failure mode — they're our infrastructure reaching in, rather than clients reaching out. I'd also have collectors emit a heartbeat so their own availability is observable, and alert on absence of expected report volume relative to a baseline rather than on error count alone.

"A region goes dark. Walk me through what you actually see."

Ideally a spike in reports from that region's autonomous systems arriving at a collector outside the affected path — that's the good case. The harder case is that the failure also blocks the collector, so we see silence from a region that normally reports steadily. That's why baseline-relative alerting matters: a region going quiet is itself the signal. And buffered reports will arrive later in a burst, timestamped during the outage, so spike detection has to key on event time rather than arrival time or the incident appears to have happened when connectivity returned.

"Why not just put the collector in a different data center?"

Because that only separates one failure domain. A different data center might still be the same domain name, the same autonomous system, and behind the same CDN — so a DNS failure, a BGP hijack, or a CDN outage takes out both. The separations are a ladder: different IP survives a host failure, different domain survives DNS, different ASN survives a hijack. You want them composed, not just geographic separation.

"Isn't sampling at 1% going to miss things?"

Not for what we're detecting. We're estimating a rate — has the client-side error rate in some population spiked — and rates survive sampling well. A regional outage affects a large fraction of that region's users, so 1% surfaces it clearly. Sampling would be wrong if we needed a specific user's session, which is the distributed-tracing case. The rule I'd apply: sample when you need a rate, keep everything when you need an instance.

"Privacy team says no client telemetry at all. What's left?"

Probing, and it becomes much more important. We'd lose real-user coverage and fall back to synthetic checks from as many vantage points as we can justify, plus anything the network layer gives us — BGP monitoring services that watch for route anomalies affecting our prefixes, which is external and involves no user data at all. I'd also lean on aggregate server-side signals as a weak backstop, while being honest that regional failures may stay invisible. It's a real degradation, not a workaround.

"Would you build this or buy it?"

Buy the probing — synthetic monitoring from global vantage points is a commodity and running your own probe fleet across many networks is exactly the cost problem we identified. The agent-and-collector side is more likely to be built, because it's coupled to your client and your consent model, though real-user-monitoring products exist. And BGP route monitoring is definitely bought — it needs a view of global routing tables that you have no reason to build.

Now do it live

The next section drills these as standalone probes.