Reaching Collectors Under Failure

In one line: an error report that fails to send is worthless. This lesson is about making the reporting path survive the exact failure it is reporting — which requires being deliberate about which failure domain the collector sits in.

Separation at every level

The pattern applies at multiple levels, because failures occur at multiple levels:

Matching each failure to the separation that survives it:

Read the table as a ladder, not a menu

Each row separates at a higher level than the one above. A different IP survives a host failure but not a DNS failure. A different domain survives DNS but not an AS hijack. A different ASN survives routing but not the user's own connection dying.

So the separations compose — a collector on a different IP, under a different domain, in a different autonomous system survives all three. Each one you skip is a specific failure class whose reports you will never receive.

And note the direction of the correlation: the more severe the failure, the higher up the ladder you need to have separated. The failures you most want to hear about are the ones that require the most separation to report.

The last mile has no answer

The final row is the honest one: for last-mile connectivity errors — where the user has no internet access at all — there is no readily available fall-back.

You cannot send a report over a network that does not exist. So:

The agent accumulates events locally and reports them once connectivity is restored.

Buffered reports arrive late and skew your timeline

Store-and-forward is the right answer, and it has a consequence worth planning for: reports about an outage arrive after it ends, in a burst, timestamped in the past.

Two implications. First, spike detection must key on the event timestamp, not arrival time, or the outage appears to have happened when connectivity returned. Second, the burst of delayed reports can look like a new incident if you are not careful.

The buffer is also finite — the same dilemma as the previous chapter's monitoring buffers. If it fills, drop the oldest: during a long outage, the most recent events describe the current state.

The report generated as the page dies

There is a delivery problem that has nothing to do with network paths: some of the most valuable reports are produced at the exact moment the client stops existing.

The correlation is what makes it matter. A crash that causes the user to close the tab is both more serious than average and less likely to be reported than average, so the data you collect is biased toward the failures people tolerated.

The fix is a send that does not depend on the page surviving: a fire-and-forget beacon the browser or OS takes ownership of and delivers after the page is gone. It is deliberately limited — small payload, no response, no retry visible to the caller — and those limits are exactly why it can be guaranteed.

Two design consequences:

Report on the way out, not only on the way in. Flush pending reports when the page becomes hidden rather than waiting for a close event, because on mobile a backgrounded app may never receive one.

Keep the exit payload small. Whatever must survive the page has to fit in a beacon, so the agent should hold a compact summary — error fingerprint, release, a short breadcrumb trail — rather than assuming it can send everything it has buffered.

That last point interacts with buffering from the previous section: a large buffer is fine while the page lives and useless the instant it does not. Buffer for retry; summarise for exit.

Why this mirrors the previous chapter

The server-side chapter ended on "who monitors the monitor?" and answered it with an external prober outside the infrastructure.

This is the same principle applied to the reporting path:

Interview signal by level

Next: what you are allowed to collect.