PII and Data Leakage

In one line: redaction is a detection problem with no complete solution, so the design has to assume some PII gets through and limit where it can travel.

Detection first, and its ceiling

The standard toolkit combines three techniques, and each covers what the others miss.

Pattern matching handles structured identifiers — card numbers, national insurance numbers, IBANs. Strong here because many have checksums, so a validated match is nearly certain.

Named entity recognition handles unstructured mentions: names, addresses, employers, dates of birth. This is where a model earns its keep and also where it fails, because "Paris" is a city or a person and only context decides.

Context rules catch what neither does alone. A bare seven-digit number is nothing; the same number after "my account is" is an identifier.

Microsoft Presidio is the reference implementation of this combination — regex recognisers plus a spaCy NER backend plus configurable context rules — and its own documentation is explicit that automated detection cannot find every sensitive value.

That last box is the lesson. Treat the residual as a design input: since some PII reaches the model, the controls that matter are the ones limiting where it can go afterwards.

Reversible tokenisation

Naive redaction destroys the request. Replace every name with [REDACTED] and the model can no longer tell two people apart, resolve a pronoun, or address the user.

The working pattern keeps a mapping:

in    "Email Sarah Chen at sarah@acme.com about invoice 4471"
send  "Email <PERSON_1> at <EMAIL_1> about invoice <INV_1>"
map   PERSON_1 -> Sarah Chen, EMAIL_1 -> sarah@acme.com, INV_1 -> 4471
out   "I've drafted a note to <PERSON_1> at <EMAIL_1>..."
show  "I've drafted a note to Sarah Chen at sarah@acme.com..."

The model reasons over stable placeholders and never receives the values. Detokenisation happens at the boundary, after every guard has run.

Three implementation details that decide whether it works:

Placeholders must be type-consistent and stable. The same person is PERSON_1 throughout, and a person is never an EMAIL. Otherwise the model's reasoning about relationships breaks.

The map is scoped to one request and never persisted with the prompt. A map that outlives its request is the leak it was built to prevent.

Anything unmapped in the output is suspect. If the model emits a token that is not in the map, either it hallucinated an identifier or it reconstructed one — both worth flagging.

The leaks that are not about the model

Most real incidents are plumbing. The model behaved correctly and repeated what it was handed.

The retrieval boundary. The single largest source. A vector index without a tenant partition returns nearest neighbours across tenants, and the model faithfully summarises somebody else's document. The fix is not a guard — it is a filter in the query path, enforced by the index rather than by a post-filter that a code path can skip.

Logs. Prompts and completions get logged for debugging, and now PII lives in a log store with a different retention policy and a much wider access list than the primary database. Log identifiers and hashes, not content, or apply the same redaction to logs that you apply to prompts.

Caches. A semantic cache keyed only on the query serves one user's completion to another whose question was similar. Cache keys for personalised responses must include the identity, and personalised responses often should not be cached at all.

Training and feedback. Conversations collected for fine-tuning or evaluation carry everything users typed. That corpus needs the same treatment as production data and usually a separate consent basis.

The system prompt. Anything in it is one successful extraction away from public. Keys, internal URLs and customer names do not belong there — not because extraction is certain, but because the prompt is the least defensible place to keep a secret.

Retention and deletion

The question that separates a design from a demo: when someone asks you to delete their data, what actually happens?

Conversation history, the vector index, the cache, the logs, the eval sets, the fine-tuning corpus. Every one is a copy, and a deletion that misses any of them is not a deletion.

Vector indexes are the awkward one. Many support only a soft delete — the vector stays until a compaction that may be scheduled weeks out — so a filter at query time is doing the work in the meantime. That is acceptable if you know it and unacceptable if you told a regulator otherwise. Know which of your stores support hard delete and which are relying on a filter.

What to say when asked

The strong answer has four parts, in this order: detect with layered techniques and state that recall is high but never complete; tokenise reversibly so utility survives; enforce the tenant boundary in the retrieval path rather than in a guard; and treat logs, caches and training corpora as first-class copies with the same policy as the primary store.

The weak answer is "we run PII detection on the input." It addresses one hop of five.

Next: the failure you will never get a complaint about.