Concept Drills: 12 Client-Side Monitoring Probes

This topic appears as a follow-up whenever availability or monitoring comes up — usually phrased as "everything's green but users are complaining."

The problem

1. Why can't server monitoring detect client-side errors? · L5 · Testing: the core asymmetry

Because a request that never arrives leaves no trace. Server-side failures show up as elevated HTTP 500s in web and application logs. But if DNS fails, or routing sends traffic elsewhere, or a middlebox drops it, there's no log line — and our infrastructure is genuinely healthy, so every internal metric is correctly green.

2. What are the three classes of client-side failure? · L4 · Testing: recall

DNS resolution failures, routing failures between the client and the service provider, and third-party infrastructure failures like middleboxes or CDNs. All three are dependencies we don't own, sitting in front of infrastructure we do.

3. Why not just alert on a drop in traffic? · Staff · Testing: killing the obvious answer

Two reasons. Traffic varies naturally — time of day, day of week, campaigns ending — so a dip-based threshold either fires constantly or is tuned too loose to catch anything. And when the issue affects only a segment of users, the aggregate barely moves: a whole country losing access might be 2% of global traffic, well inside normal variance. The signal is a small population vanishing from a large one, which is exactly what aggregates hide.

4. Explain the BGP leak case. · Staff · Testing: the concrete example

A peer ISP mistakenly announced incorrect BGP routes for a major provider, redirecting customer traffic through unintended networks so it never reached its destination. Users couldn't access the service and the provider didn't immediately know, because the outage originated outside its own infrastructure. And these aren't rare — in April 2021 an AS mis-announced over 30,000 prefixes, causing a 13x inbound traffic surge that network operators, not the affected services, detected and mitigated.

5. What's the general principle behind all of this? · Staff · Testing: transfer

To detect a failure in a layer, you have to observe from outside that layer. It's failure independence applied to observation rather than redundancy: a monitor that shares the failure can't report it. Same reasoning as the server-side chapter's external prober.

Probing

6. How would probing work, and what does it get you? · L5 · Testing: the first design

Deploy probers that periodically send requests to the service from various global vantage points, so you're testing reachability from the client side of the network path. That lets you distinguish "unreachable from Frankfurt" from "globally down" — a real external signal rather than an inference from traffic volume.

7. Why isn't probing sufficient? · Staff · Testing: the ceiling

Two limits. Coverage — the internet has over 100,000 autonomous systems, and running probes in every one is cost-prohibitive and logistically hard given varying regulations and maintenance. A few hundred vantage points is a vanishing sample, and the failures that matter are concentrated in networks we have no probe in. Synthetic traffic — probes don't represent real user behavior or complex interaction patterns. Probing gives presence, not coverage.

8. So do you drop probing? · L5 · Testing: fairness

No. It's cheap, needs no user consent, works before you have any users in a region, and gives an unambiguous external signal — which is exactly what you need to answer "is anything alive?" when agent reports go quiet. It's complementary to agent reporting, not superseded by it.

The design

9. Describe the agent-and-collector design. · L5 · Testing: the mechanism

An agent embedded in the client application sends a report when a request fails, and a collector — a standalone service independent of the primary service — receives those reports so we can detect spikes. The independence is the point: if reports went to the main service, they'd fail for the same reason as the original request.

10. How do you scale and afford it? · Staff · Testing: the supporting decisions

Collectors deployed close to client networks to cut latency, organized in a hierarchical aggregation architecture, with stream processing for near-real-time analysis. And sampling — around 1% of users rather than every event, since ingesting everything would need substantially more infrastructure. Sampling works because we're estimating a rate, not reconstructing individual sessions.

11. Where do you put the collector, and why? · Staff · Testing: failure domains

Outside the blast radius, separated at every level that can fail: different IP so a host failure doesn't take it, different domain so a DNS failure doesn't, different autonomous system so a BGP hijack doesn't, and different or no CDN. Those compose — each one skipped is a failure class we go blind to. For last-mile loss there's no fall-back at all, so the agent accumulates events locally and reports once connectivity returns.

12. What can't you collect, and what's the test? · Staff · Testing: privacy as a constraint

No traceroute, because hops reveal geographic location. No DNS resolver details, same reason. And limited RTT or packet-loss data if it risks identifying the user. Notice every excluded field is excluded because it's diagnostic — they're the same information. The test I'd use: would this field already appear in our weblogs on a successful request? If yes it's safe; if no, a failure isn't a licence to collect it. Plus end-to-end encryption — not for secrecy but for tamper-resistance, since the intermediaries causing these failures shouldn't be able to suppress reports about themselves.

Self-check

The cheat sheet next compresses the chapter onto one page.