Cheat Sheet

Why it's invisible

Three failure classes: DNS resolution failures · routing failures between client and provider · third-party infrastructure (middleboxes, CDNs). All are dependencies you don't own, sitting in front of infrastructure you do.

Why a traffic dip doesn't work:

• Natural variability — dips are normal, so the alert either spams or catches nothing
• Partial impact — a whole country going dark may be ~2% of traffic, inside normal variance

The signal is a small population vanishing from a large one — exactly what aggregates hide.

The BGP leak

A peer ISP announced incorrect BGP routes, redirecting customer traffic through unintended networks. Users couldn't reach the service; the provider didn't know, because the outage originated outside its own infrastructure.

April 16, 2021: an AS mis-announced more than 30,000 BGP prefixes → 13x inbound traffic surge. Network operators detected and mitigated it — not the affected services.

Principle: to detect a failure in a layer, observe from outside that layer. (Failure independence, applied to observation.)

Active probing

Prober — a service that periodically sends requests from global vantage points to check reachability. External vantage point ✓.

Limits:

• Incomplete coverage — the internet has > 100,000 autonomous systems; probing every one is cost-prohibitive and logistically hard (regulations, maintenance)
• Synthetic traffic — doesn't represent real user behavior or complex interaction patterns

Probing gives presence, not coverage. Keep it: cheap, no consent needed, works with zero users, and it's the independent signal when agent reports go quiet.

Agent and collector

Scaling: collectors close to client networks (latency) · hierarchical aggregation · stream processing for near-real-time · sampling ~1% of users to control cost.

Why sampling is safe here: we're estimating a rate, not reconstructing sessions. Sample when you need a rate; keep everything when you need an instance.

Coverage advantage over probing: reports come from every network with a real user in it.

Consent and control

Custom HTTP header signals the client supports reporting — sent only if the user consented. The service replies with policy and collection endpoint values.

• Endpoint delivered as data, so collectors can move without shipping a client
• Needs browser support on the web — feasible for orgs that make browsers (Chromium-based) and can standardize over time
• A client-side application you control makes this trivial
• Consent must be revocable at any time, not a one-time flag

"Do we control the client?" changes the answer materially — it buys both capability and standing to ask for more data.

Reaching collectors under failure

Collectors must reside in a different failure domain — outside the "blast radius."

The separations compose — each one skipped is a failure class you go blind to. Geographic separation alone is not enough (same domain, same ASN, same CDN).

Buffered reports arrive late, in a burst, timestamped in the past → key spike detection on event time, not arrival time. Buffer is finite: drop the oldest.

Privacy

Users must have full control over what is collected and the ability to opt out at any time.

Every excluded field is excluded because it's diagnostic — they're the same information. So minimize rather than anonymize.

The test:

Would this field already appear in our weblogs
on a SUCCESSFUL request?

  yes -> safe to report on failure
  no  -> a failure is not a licence to collect it

• No active probing beyond standard service functionality
• End-to-end encryption — for tamper-resistance, so intermediaries can't suppress reports about their own faults
• Collect the minimum necessary, use it only for the consented purpose (no repurposing for analytics)
• Custom apps have more latitude than browsers, because the consent relationship differs

Quick decision cues

• Dashboards green, users complaining → client-side, look outside
• Need an external signal with no users yet → probing
• Need real coverage across networks → agent reports
• Agent reports went to zero → could be healthy or collector unreachable — check probers
• Collector placement → different IP + domain + ASN, composed
• User has no connectivity → buffer and forward, key on event time
• Browser client → strict minimization; native client → more latitude
• Deciding what to collect → weblog-equivalence test
• Privacy team blocks telemetry → fall back to probing + BGP route monitoring

Work the Interview Walkthrough for the full design and the Concept Drills for rapid-fire practice.