Reaching Collectors Under Failure
Why this matters: an error report that fails to send is worthless. This lesson is about making the reporting path survive the exact failure it is reporting — which requires being deliberate about which failure domain the collector sits in.
Key takeaway
Collectors must reside in a different failure domain than the monitored service. If the main service fails, the collector must remain accessible — keeping the collector outside the "blast radius" of a fault.
Separation at every level
The pattern applies at multiple levels, because failures occur at multiple levels:
| Level | Separation required |
|---|---|
| IP reachability | Host the collector on a different IP address |
| Domain availability | Use a different domain for the collector |
| AS hijacking | Host the collector in a different autonomous system |
Matching each failure to the separation that survives it:
| Failure | What the collector needs |
|---|---|
1.2.3.4 unreachable | Different server IP |
Can't resolve example.com | Different domain |
| AS 1234 hijacked | Different ASN |
| CDN unavailable | Different or no CDN |
| Last-mile problems | No readily available fall-back for the service |
The last mile has no answer
The final row is the honest one: for last-mile connectivity errors — where the user has no internet access at all — there is no readily available fall-back.
You cannot send a report over a network that does not exist. So:
The agent accumulates events locally and reports them once connectivity is restored.
Why this mirrors the previous chapter
The server-side chapter ended on "who monitors the monitor?" and answered it with an external prober outside the infrastructure.
This is the same principle applied to the reporting path:
| Server-side monitoring | Client-side monitoring | |
|---|---|---|
| The risk | Monitoring shares fate with the monitored infrastructure | The error report shares fate with the failed request |
| The symptom | Silence is ambiguous — healthy, or dead including the observer? | No reports arrive — no errors, or no way to report them? |
| The fix | An observer outside the infrastructure | A collector outside the failure domain |
Key takeaway
Separate the collector by IP, domain, and autonomous system — the separations compose, and each one skipped is a failure class you will never hear about. For the last mile, buffer locally and forward on recovery, keying analysis on event time rather than arrival time.
Interview signal by level
| Level | What a strong answer sounds like |
|---|---|
| L4 | "The client sends the error report to our servers." |
| L5 | Separates them: "the collector has to be on separate infrastructure, or the report fails for the same reason the request did." |
| Staff+ | Ladders the separation and handles the last mile: "different IP survives a host failure, different domain survives DNS, different ASN survives a hijack — they compose, and each one skipped is a failure class we go blind to. For last-mile loss there's no fall-back, so buffer and forward, and key spike detection on event time or the outage looks like it happened when connectivity came back." |
Next: what you are allowed to collect.