The Camera That Stopped Recording

In one line: everything can look healthy while the only thing the system exists to do has quietly stopped.

Why this failure is different

In a request-serving system, failure announces itself. Errors rise, latency spikes, someone complains. The feedback loop is minutes long.

Here, nobody is watching. A camera that stops recording produces no errors for anyone to see, no user complaints, and no change in any dashboard that measures requests — because there were never any requests. The absence of footage is indistinguishable from footage of nothing happening.

The loop closes weeks later, when there is an incident, someone opens the timeline, and the footage is not there. At that point the system has failed at its only job, and failed in the specific circumstance it was bought for.

Measure continuity, not liveness

The usual health check asks whether the camera is reachable. That is necessary and badly insufficient — a camera can respond to every ping while sending no video, sending video that fails to write, or sending video with a clock two hours off.

The right signal is derived from the control plane, and it is cheap because those events are already flowing. For each camera you know the expected segment cadence, so you know how many segments should have arrived in the last hour:

expected  = 3600 / target_segment_seconds
observed  = count(segments where camera=C and capture time in last hour)
continuity = observed / expected

Anything materially below 1.0 is a recording gap, whatever the camera's ping status says. This one derived metric catches the entire class: dead camera, wedged encoder, failing uplink, full local disk, gateway dropping writes, storage rejecting puts.

The failure modes underneath

The last row is honest and worth saying. A camera pointed at a wall records flawlessly. No infrastructure metric will ever flag it, and the only detection is scene-change analytics or a human periodically confirming that each camera still shows what it is supposed to. Acknowledging a failure mode your monitoring genuinely cannot catch is a stronger answer than pretending the design is complete.

Gaps are data

When footage is missing, the system should know and say so. A gap record — camera, start, end, cause if known — turns an invisible absence into a visible fact.

Gap records do three jobs. They render on the playback timeline, so an operator sees a marked outage rather than a seamless jump. They feed the continuity metric. And they close the loop on backfill: when a camera reconnects and uploads Tuesday's buffered footage, the gap narrows or disappears, which is exactly the confirmation an operator needs.

The reconnect storm

One capacity failure specific to this shape. Restart a gateway holding two hundred cameras and all two hundred reconnect at once — and each may then push buffered footage, so the surge is both connections and a burst of backfill traffic well above steady state.

Two mitigations, both cheap. Jittered exponential backoff on the camera or edge side spreads reconnects over a window. Rate-limiting backfill separately from live ingest keeps recovery from starving the cameras still recording normally — live footage is more important than footage from two hours ago, and the system should say so explicitly rather than letting them compete.

What to verify, and how often

The uncomfortable question an interviewer may ask: how do you know the footage is actually there and playable?

Continuity proves segments were indexed. It does not prove the bytes are readable. A periodic sampling audit closes that: pick a small random sample of segments per camera per day, fetch them, verify the checksum recorded at ingest, and confirm the container parses. At a few segments per camera per day this is negligible load, and it is the only thing that detects silent corruption or objects that were indexed but never successfully written.

Next: the sensitivity of the thing you have just built a petabyte of.