Governance and Serving
In one line: somebody will eventually ask what a specific person could see on a specific date, and the system has to be able to answer.
The audit requirement
Not logging for debugging — a record that stands up to a compliance review or a legal request.
Three questions it must answer, and each implies a different record:
What did this person search for? Every query, with the identity, the timestamp and the surface.
What were they shown? The results returned and the answer generated, since "what could they see" is not the same as "what did they look at".
Why were they permitted to see it? The permission set at the time of the query and the ACL that matched. This is the one people forget, and it is the one that makes an audit conclusive rather than suggestive — reconstructing a historical permission decision from current data is impossible, because both sides have moved.
Two design consequences. Retention for these logs is set by policy rather than by engineering convenience, often years. And the audit log is itself sensitive — a record of every query every employee ran is a serious disclosure if mishandled, so it needs stricter access control than the content index.
That second point is worth volunteering. An audit trail built to protect the organisation becomes a surveillance dataset if it is not itself governed.
Residency and isolation
Constraints that come from outside engineering and reshape the architecture.
Data residency. Content from a region may be legally required to stay in it, which means the index is regional rather than global — and a query from someone entitled to see documents in two regions must federate across two indexes rather than searching one.
Tenant or affiliate isolation. Some organisations require hard separation between business units — not a filter, but separate infrastructure, because the compliance requirement is about the data not being co-located.
Retention and deletion. A document deleted at source must disappear from the index, its embeddings, its caches and any generated answer that quoted it. That last one is the awkward member of the list: a cached answer citing a deleted document is a copy of content that was supposed to be gone.
The deletion path deserves the same treatment as in any system holding copies: enumerate every store that holds derived data and confirm each one can actually delete rather than soft-delete. Vector indexes are frequently the weak link, since many support only a tombstone until compaction.
The request path
Where the latency goes, and there is one stage here that no other search system has.
| Stage | Note |
|---|---|
| Identity resolution | Cached, and it is on the critical path |
| Query understanding | Cheap |
| Retrieval | Lexical and vector, ACL in the predicate |
| Reranking | Cross-encoder over a shortlist |
| Permission verification | Late binding, on the results being returned |
| Generation | Dominant if answering |
| Audit write | Asynchronous |
Identity resolution is the stage unique to this design, and it must be a cache hit in the normal case. A cold resolution walking a nested-group graph in the request is a tail-latency problem and a dependency on the directory being up.
The corresponding failure question: what happens when the directory is unavailable? Serving from a stale cached permission set is a security decision, not a performance one, and it needs an explicit bound — serve from cache up to some age, then fail closed. Failing closed means the product is down, and that is the correct trade here, unlike a consumer system where availability usually wins.
Degradation
With that constraint stated, the rest of the ladder is ordinary.
Full pipeline with generation. Then retrieval with no generated answer — a ranked list still works and is what the product was before. Then lexical-only if the vector index is unavailable, which loses vocabulary matching and remains useful. Then a clear error.
Two rules that do not bend. The ACL predicate never degrades — there is no rung where permissions are skipped for speed. And identity resolution never degrades to a permissive default; an unknown or unresolvable principal fails closed.
Cost
Briefly, because the shape differs from consumer systems.
The corpus is small by web standards — a million documents is unremarkable — so index storage and retrieval are inexpensive. Two things dominate instead.
Connector operation, which is ongoing engineering time rather than compute, and it does not go away.
Generation, if every query produces an answer. At a few thousand queries a day this is modest in absolute terms, and it is the largest line in the serving bill.
Which produces a useful conclusion: at this scale, engineering time is the dominant cost, not infrastructure. The design choices that matter are the ones that reduce ongoing maintenance — buying commodity connectors, keeping the ranking interpretable rather than learned, avoiding infrastructure that needs a specialist to operate.
That is a genuinely different optimisation target from every other chapter in this course, and it is worth naming as the closing thought.
Key takeaway
Log the permission decision with its inputs, or an audit question about a past date is unanswerable — and govern the audit log itself, since a record of every employee's queries is a surveillance dataset. Residency makes the index regional and federated. Identity resolution is on the critical path and must be a cache hit, and when the directory is unavailable this is the one system in the course that should fail closed. At this scale the dominant cost is engineering time, not infrastructure.
Next: the whole thing, as an interview.