Restraining Log Volume: Sampling and Categorization
In one line: every logging system eventually confronts the same question — you cannot afford to record everything, so what do you drop? The two answers drop very different things, and one of them is unsafe for some workloads.
Sampling
Sampling reduces log volume by recording a representative subset of events. In a social network where users generate high volumes of comments, logging every interaction is impractical. A sampling service captures a configurable percentage of these events, enabling monitoring of system health and trends without overwhelming storage or processing resources.
Note: for hyperscale systems like Facebook, logging billions of events per second is not viable. An appropriate sampling threshold and strategy are necessary to selectively pick a representative data set.
Sampling works for statistics and fails for individuals
The distinction that decides whether sampling is safe:
Aggregate questions survive sampling. "What's our p99 latency?", "Is the error rate rising?", "Which endpoint is slowest?" — a 1% sample answers all of these nearly as well as 100%, because you are estimating a distribution and statistics are robust to sampling.
Individual questions do not. "What happened to request abc123?", "Why did this specific payment fail?", "What did this attacker do?" — a 1% sample means a 99% chance the record you need was discarded.
So sampling is not a percentage-of-fidelity knob; it is a change in what questions you can answer at all. It preserves trend analysis and destroys forensics.
Which is exactly why Lesson 1's four consumers conflict: analytics is happy with samples, breach response is not. Sample the aggregate pipeline, keep the forensic one complete.
Where sampling breaks entirely
"What is a scenario where the sampling approach will not work?"
Consider an application that processes an ATM financial transaction. It runs various services — fraud detection, expiration time checking, card validation. If we start missing service logs, we cannot identify an end-to-end flow, which affects debugging when an error occurs. Sampling here is not ideal and results in the loss of useful data.
The reason this is worse than losing 99% of records is worth spelling out: naive sampling breaks traces, not just events.
A single transaction touches fraud detection, expiry check, and validation. Sample each service's logs independently at 10%, and the probability that all three survive is 0.1 × 0.1 × 0.1 = 0.1%. You end up with a large collection of fragments — a validation log with no fraud-check log, an error with no preceding context — which is arguably worse than no logs, because it looks like evidence and misleads.
The fix, when you must sample traced workloads, is consistent sampling: decide once per request — usually at the frontend, based on the trace ID — whether to log it, and propagate that decision to every service. Then you keep complete traces for 10% of requests rather than 10% of the events in every trace.
Sample requests, not events. That single reframing is what makes sampling compatible with tracing, and it is a strong thing to volunteer.
You can also categorize message types and apply filters to log only high-priority events.
Categorization by severity
Standard logging libraries — Log4j for Java, the logging module in Python — use severity levels to categorize messages:
| Level | Meaning |
|---|---|
DEBUG | Detailed information for diagnosing problems |
INFO | Confirmation that things are working as expected |
WARNING | Indication of a potential problem or unexpected situation |
ERROR | A serious issue preventing a specific function from executing |
FATAL/CRITICAL | A severe error causing the program to crash |
In production, logs are typically filtered to capture WARNING levels and above to reduce noise. However, DEBUG and INFO levels are essential during development or when troubleshooting complex flows.
The Python example, setting the threshold to DEBUG so every level prints:
import logging as log
# set the logging level to DEBUG
log.basicConfig(level=log.DEBUG)
for i in range(6):
if i == 0:
log.debug("Debug level")
elif i == 1:
log.info("Info level")
elif i == 2:
log.warning("Warning level")
elif i == 3:
log.error("Error level")
elif i == 4:
log.critical("Critical level")Output:
DEBUG:root:Debug level INFO:root:Info level WARNING:root:Warning level ERROR:root:Error level CRITICAL:root:Critical level
Severity filtering and sampling drop opposite things — that's why you need both
These are often lumped together as "reduce log volume." They are not the same operation:
Severity filtering drops whole categories — every DEBUG message, always. It is deterministic: you know exactly what you kept.
Sampling drops a random fraction of all categories. It is probabilistic: you kept a representative slice of everything.
The failure modes differ accordingly. Severity filtering means you have complete ERROR records and zero DEBUG context — you know something broke but not the sequence leading to it. Sampling means you have partial records of everything.
Production usually wants both: filter by severity to kill the bulk, then sample what remains if it is still too much. And the standard escape hatch is dynamic log levels — the ability to raise a single service to DEBUG at runtime while investigating, without a deploy. That is the practical answer to "we filtered out exactly the context we now need."
Severity levels are a social contract, and they decay
The definitions above are crisp. In practice teams disagree — one engineer's WARNING is another's INFO, and levels drift upward over time as people escalate their own messages to make sure they are seen.
The predictable result is ERROR fatigue: so many ERROR entries that nobody reads them, at which point the level has stopped carrying information. Lesson 8's alert aggregator fires on fatal errors, so mislabelled severity translates directly into either missed incidents or alert spam.
The practical discipline: ERROR should mean "a human should look at this", and if that produces more entries than a human can look at, the labelling is wrong rather than the volume. Worth saying, because it is an operational failure interviewers recognize immediately.
Key takeaway
Sampling preserves aggregate statistics and destroys forensics — and applied naively per service it shatters traces, so sample per request and propagate the decision. Severity filtering drops whole categories deterministically, typically keeping WARNING and above. They drop opposite things, so production uses both, plus dynamic log levels for when you filtered out the context you now need.
Interview signal by level
| Level | What a strong answer sounds like |
|---|---|
| L4 | "Log only warnings and above, and sample if there's still too much." |
| L5 | Knows where sampling fails: "sampling is fine for trends but not for tracing a specific transaction — for a payment flow you need the complete end-to-end path, so you can't drop service logs." |
| Staff+ | Fixes sampling rather than rejecting it: "sampling independently per service shatters traces — three services at 10% each gives complete traces 0.1% of the time, and fragments are worse than nothing because they mislead. So sample per request: decide once at the frontend from the trace ID and propagate it, keeping complete traces for 10% of requests. And I'd pair severity filtering with dynamic log levels, so we can raise one service to DEBUG at runtime rather than discovering we filtered out the context we need." |
Next: the shape of a log line, and what must never be in it.