Answering, Not Finding

In one line: a result list says "this might help"; an answer says "this is true", and only one of those can be wrong in a way that matters.

What changes

A ranked list transfers the work and the judgement to the reader. They open the document, see its date and author, notice it is from a team that no longer exists, and discount it.

An answer removes all of that. The system has read the documents, decided which are relevant, resolved any conflicts between them, and stated a conclusion. The reader sees a fluent paragraph.

That is a real product improvement and a real increase in liability. The specific enterprise version: a fluent, cited, incorrect answer about parental leave, expense policy, or a security requirement is worse than no answer, because the employee acts on it.

Abstention is a feature

The design consequence, and it is the thing to say first.

"I could not find a confident answer, here are the closest documents" is a good output. It is honest, it degrades to the thing that already worked, and it does not manufacture certainty.

Most designs treat generation as unconditional — retrieve, generate, return. Making the answer conditional on the retrieval being good enough is a decision worth stating.

Three practical triggers for abstaining:

Weak retrieval. The top results score poorly, or nothing clears a similarity floor. There is nothing to ground an answer in.

Disagreement among sources. Two retrieved documents say different things. That should surface as a conflict, not be silently resolved by whichever the model weighted more.

A question the corpus does not cover. Asking about something the organisation has never documented should return "we do not appear to have this written down", which is a genuinely useful answer and an input to the documentation backlog.

Attribution has to be per claim

Citing three documents at the end of a paragraph is not attribution — the reader cannot tell which sentence came from which source, so they cannot verify anything without reading all three.

Per-claim citation, where each assertion links to the specific passage supporting it, is what makes verification possible. And verification matters more here than in a consumer product, because the answers are used to make decisions with consequences.

The mechanical check from the RAG machinery applies directly: verify that every cited passage exists, that it was actually retrieved for this request, and that it is textually similar to the sentence citing it. That catches fabricated citations, which are common enough to be worth a cheap deterministic check rather than trust.

Conflict is the enterprise-specific case

The failure that does not occur in a consumer knowledge assistant, and it follows directly from the staleness problem.

Retrieval returns five documents about the deployment process. Three are superseded. The model reads all five and produces a fluent synthesis that blends the current process with two obsolete ones — internally coherent, well cited, and describing a process that has never existed.

Three responses, in increasing order of quality.

Detect and surface it. If retrieved passages disagree on a fact, say so and show both with their dates. Cheap, honest, and it puts the judgement back where it belongs.

Prefer the canonical. Where lifecycle metadata exists, weight it heavily in assembly rather than only in ranking — the current version should dominate the context, not merely lead it.

Filter by recency before generating, on a per-topic basis. Risky, since older material is sometimes the right answer, but appropriate for categories like policy where currency is definitional.

The permission consequence of answering

A subtlety specific to generation, and it is worth raising because it sounds like a problem and mostly is not.

If retrieval is permission-filtered correctly, every document in the context is one the user may read, so an answer synthesised from them reveals nothing they could not have read themselves.

The residual risks are narrower and real:

Synthesis across documents can reveal something no single one states. Two permitted documents combining into an inference the user was not meant to draw. Rare, and mostly unavoidable — they could have performed the same inference by reading both.

Caching answers across users is a breach. An answer generated for one person's permission set must never be served to another. Any cache of generated answers keys on the permission set, or it does not exist.

That second one is the practical hazard, and it is the same rule as the personalised-cache rule elsewhere: anything holding permission-dependent output keys on identity.

Feedback, which is the only signal you get

The closing point, and it ties back to the ranking problem.

With no click data, explicit feedback on answers is disproportionately valuable. A thumbs-down on a generated answer is worth far more than a click, because it is unambiguous and it arrives on exactly the cases that failed.

Two design points. Ask for a reason — wrong, outdated, not permitted to see this, incomplete — because the category routes the fix to ranking, to the documentation owner, or to the permission model. And close the loop: an answer marked outdated should notify the document's owner, which turns the search system into something that improves the corpus rather than only reading it.

That last idea is worth volunteering. The system sits on top of the organisation's knowledge, so it is uniquely placed to tell the organisation what is missing, stale or contradictory — and that may be more valuable than the search itself.

Next: evaluating a system with no traffic.