Evaluation Without Traffic

In one line: you cannot measure this the way a consumer product is measured, so the measurement has to be built rather than collected.

Why the usual instruments fail

A/B tests need power you do not have. Detecting a modest ranking improvement takes far more queries than a few thousand a day supplies, and the queries are heterogeneous — a handful of common lookups and a long tail of unique ones. A two-week test resolves only catastrophic regressions.

Click-through is a weak and misleading signal. A user who clicks three results and leaves may have found what they needed on the third, or given up. In a corpus where the right answer is one specific document, clicks say much less than they do on the web.

There is no external benchmark. Your corpus, your policies and your organisation's vocabulary are unique. No public dataset measures whether this system works for these people.

So the evaluation is constructed, and it has three parts.

The judged set

The core artifact, and building it well is most of the work.

Where the queries come from. Real logs are the best source, plus interviews with people across teams about what they actually try to find. Include the questions that have never worked, because those are the ones you are trying to fix.

Stratify rather than sample by traffic. Traffic is dominated by a few common lookups. A traffic-weighted set will be nearly all easy queries and every regression on the long tail will be invisible — the same trap as head-weighted evaluation in search ranking.

Stratify across: query type (known-item, exploratory, question), source system, team, and difficulty.

Who judges. People who know the domain. A generic annotator cannot tell which of five deployment documents is current, which is precisely the judgement that matters here. This makes the set expensive and it is the reason it must be small and well-chosen.

Graded, not binary. The distinction between the ideal document and an acceptable one is the whole of ranking quality.

Re-judge periodically. The corpus moves and judgements go stale. A judgement referencing a superseded document is now wrong in a way that penalises correct behaviour.

The permission test suite

The part that is not about quality at all, and it deserves separate treatment because it is a correctness requirement.

Construct test users with known, deliberately varied permission sets, and a set of documents whose visibility is exactly known. Then assert, on every deploy:

• Each test user sees exactly what they should
• Result counts and facet counts do not reveal anything they cannot see
• Snippets never contain content from documents they cannot open
• A user removed from a group loses access within the stated window
• An unmapped or unknown principal fails closed

The most valuable case in that list is the fourth. It is the one that exercises the sync path rather than the query path, and it is where the real bugs live — the ACL that did not propagate, the cached expansion that outlived the change.

Evaluating answers

Beyond retrieval, the generation half needs its own checks, and they are mostly mechanical.

Grounding. Every claim traceable to a retrieved passage. Deterministic where citations are structured, and it catches fabrication cheaply.

Citation validity. Each cited passage exists, was retrieved for this request, and is textually similar to the sentence citing it.

Abstention correctness. Two rates, and both matter: how often it abstains when it should have answered, and how often it answers when it should have abstained. Reporting only the second optimises toward a system that never answers.

Conflict handling. Given a query where the corpus genuinely disagrees, does the answer surface the disagreement or silently pick one? This needs deliberately constructed cases, because it will not appear reliably in a random sample.

The experiment that does work

Not A/B testing — asking the person who just asked the question.

An in-product feedback control on every answer, with a reason category, produces a signal that is unambiguous, arrives on exactly the failing cases, and needs no statistical power because it is not a comparison.

At this scale that is worth more than any offline metric, and the reason categories are what make it actionable:

That last row is the one to highlight. A question the corpus cannot answer is not a search failure — it is a gap in the organisation's knowledge, and the search system is the only thing positioned to notice it.

What to watch continuously

Six signals, and only two are about ranking.

Zero-result and abstention rates, per team and per source. A rising abstention rate in one area usually means a connector broke, not that the questions got harder.

Index freshness per source — the age of the newest document from each connector. The alarm that catches a silently failing sync.

Permission verification divergence — how often late-binding verification removes a result early binding permitted. A direct measure of ACL staleness, and the only honest one.

Query latency, split by identity resolution, retrieval and generation.

Repeat queries by the same user in a session, which is the closest available proxy for "did not find it".

Feedback volume and mix, which is the only direct quality signal.

The honest position

Worth saying plainly, because interviewers respect it and it is true.

At this scale you cannot prove the system is good. You can build a judged set that catches regressions, a test suite that makes permission correctness non-negotiable, and a feedback loop that surfaces failures fast. That is a considerably weaker guarantee than a consumer product gets, and pretending otherwise — by quoting an offline metric as though it settled the question — is the failure to avoid.

Next: audit, residency and serving it.