Structuring Logs and What Not to Log

In one line: the Log4Shell section at the end is the most important thing here. Your logging library is production code with production privileges, and in 2021 it was the most exploited software on earth.

Structured logs

Unstructured
------------
2024-06-01 14:22:31 User 4821 failed login from 10.2.3.4 after 3 attempts

Structured
----------
{
  "ts": "2024-06-01T14:22:31Z",
  "level": "WARNING",
  "event": "login_failed",
  "user_id": 4821,
  "source_ip": "10.2.3.4",
  "attempts": 3,
  "trace_id": "abc123"
}

Note: the structuring of logs is a dense topic in itself. The source refers interested learners to the PhD thesis by Ryan Braud, "Query-based debugging of distributed systems."

Structure is what turns logs from text into data

The free-text line is readable by a human and nearly useless to a machine. Ask "how many failed logins came from this IP in the last hour?" and you need a regex that breaks the moment someone rewords the message.

The structured version is queryable: filter on event, group by source_ip, sum. The message format can change without breaking every consumer, because consumers read fields, not positions in a sentence.

Three things follow:

Interoperability. The producer and consumer agree on a schema rather than on prose. A new consumer needs no new parser.

Indexing gets easy. Lesson 5's log indexer feeds the Distributed Search building block — and the inverted-index pattern showed that indexing operates on documents with fields. A JSON log line is that document; free text has to be parsed into one first.

Trace correlation becomes a field lookup. Lesson 9's request ID as a first-class field means "give me everything for this request" is an indexed query rather than a grep.

The one-line version: structured logs are the difference between searching logs and querying them.

Points to consider

Logging must balance utility, security, and performance. Logs should contain relevant context without exposing sensitive data.

Data that reaches logs is extremely hard to remove — that's why the rule is 'never', not 'carefully'

"Do not log PII" sounds like ordinary hygiene. The reason it is absolute is what happens after the mistake.

A password logged once is now in: the local file on the node, the pub-sub tier, blob storage, every replica of that blob storage, the search index built over it, backups, and any downstream analytics copy. Lesson 8's pipeline is deliberately a fan-out, and it fans out the mistake too.

Removing it means finding and purging every one of those copies, including immutable archives and an index built from the original text. In practice that is a multi-week incident, and under GDPR-style regimes it is a reportable one.

Two consequences worth stating:

Prevention is the only real control. Redact at the producer — before the log line is created — not downstream, because downstream is already too late for something.

Treat log stores as production-sensitive. They contain a plaintext narrative of everything the system did, so access to logs is close to access to the system. The instinct to make logs widely readable "so everyone can debug" is exactly backwards for anything touching user data.

'Logging is I/O-heavy' is the constraint that shapes the whole architecture

This one line drives the biggest design decision in the chapter.

Logging is a write to disk or network on every logged event, in the middle of serving a user request. Do it synchronously and every request pays disk latency — and under load, when you are logging most, you slow down most. Logging becomes an amplifier of the incident it is supposed to explain.

That is why Lesson 6's non-functional requirement is "it must not block the application's critical path," and why Lesson 7 sends logs asynchronously via a background worker with in-memory buffering.

The cost of that choice is honest and worth naming: buffered logs can be lost if the process dies before flushing — precisely when you most want them. Lesson 7 covers the trade.

Vulnerability in logging infrastructure

In November 2021, a zero-day vulnerability known as Log4Shell (CVE-2021-44228) was disclosed in Log4j, a widely used Java logging framework. It received a CVSS severity score of 10.0, the highest possible rating. It allowed remote code execution, enabling attackers to compromise affected systems. This underscores the need to secure logging infrastructure and manage third-party dependencies carefully.

Why Log4Shell is the perfect cautionary tale for this chapter

The mechanism is worth understanding, because it makes the lesson concrete.

Log4j supported interpolation in log messages — a logged string containing a special expression would be evaluated, including expressions that fetched and executed remote code. So logging a string was not inert; it was execution.

Now combine that with what logging does: applications log user-controlled input constantly — usernames, user agents, search queries, HTTP headers. An attacker only had to get a crafted string logged. Putting it in a User-Agent header was enough.

Three lessons, all specific to this chapter:

Logging is not a passive observer. It runs code, in your process, with your privileges, on attacker-controlled input. That is a dangerous combination, and the "logging is just diagnostics" mental model is what made it invisible.

Log input is untrusted input. Anything reaching a log line came from somewhere, often from a user, and deserves the same suspicion as any other input.

Ubiquity is the amplifier. Log4j was in nearly every Java application, most of them not knowing it was there transitively. A vulnerability in something universal and unglamorous has a blast radius nothing else matches.

The takeaway for a design interview: when asked about logging security, most candidates say "don't log passwords." Saying "and the logging library itself is attack surface — Log4Shell was a 10.0 because logging executes on untrusted input" is a different level of answer.

Interview signal by level

Next: what we're building.