Ranking Without Signal

In one line: web search learns what is good from a billion clicks and a link graph, and you have neither, so you build the signals yourself.

What you cannot do

You cannot train a learned ranker. A few thousand queries a day, most of them unique, does not produce enough labelled pairs. Attempting it produces a model fitted to noise that is worse than a tuned baseline and much harder to reason about.

You cannot A/B test. Detecting a ranking improvement needs statistical power that traffic does not supply. A two-week test on a few thousand queries will not resolve anything but a catastrophic regression.

You have no link graph. Nothing confers authority. A document nobody has ever opened looks identical to the canonical policy everyone relies on.

So the answer is a hand-tuned combination of interpretable signals, evaluated against a judged set — which sounds like a retreat and is the correct engineering decision at this scale.

The signals you do have

Enterprise data carries signals web pages do not, and they substitute for what is missing.

Field matching. A term in a title, a heading or a filename is worth much more than one in the body. BM25 across weighted fields does a great deal of work here, and it is often most of the quality.

Recency, applied carefully. Newer is usually better in a corpus full of superseded material — but recency must not outrank correctness. An over-tuned recency boost pushes a new but irrelevant document above the canonical policy page, which is the classic failure here.

Author authority. Not links, but role. A policy written by the team that owns the policy outranks one written by someone summarising it. The org chart supplies this.

Proximity to the asker. Documents from the user's own team, project or space are more likely to be what they meant. This is the strongest personalisation signal available and it is nearly free.

Usage. Views, edits, shares and references from other documents. Weak individually, and it is the closest thing to a popularity signal on offer.

Canonical status. Explicit metadata — this page is the current version, this space is official. Where it exists it dominates everything else, and where it does not, creating it is often higher-value than any ranking work.

Personalisation from org structure

The signal worth expanding on, because it is specific to this problem and easy to under-use.

The same query means different things to different people. "Deployment process" from a backend engineer, a data scientist and a salesperson should return three different documents — and the org chart already knows which team each belongs to.

Useful proximity features: the document's owning team versus the user's team, distance in the reporting structure, shared project membership, and whether the user has opened or edited anything nearby.

Two cautions. Do not personalise so hard that shared knowledge becomes invisible — someone asking outside their area should still find the canonical answer. And do not confuse permission with relevance: a user may be entitled to see everything and still want their own team's version first, which is a ranking decision rather than an access one.

Tuning by hand, responsibly

Hand-tuned does not mean arbitrary, and the process is what makes it defensible.

Build a judged set first. A few hundred realistic queries, each with graded relevance judgements from people who know the domain. Stratified across teams, query types and sources — not sampled by traffic, which is dominated by a handful of common lookups.

Change one weight at a time and measure. Slow, and it is the only way to know which change did what.

Keep a regression suite. Queries with known-correct answers that must never break, drawn from complaints and from the searches that matter most.

Re-judge periodically. The corpus moves. A judgement from a year ago may reference a document that has been superseded.

That judged set is the single highest-value artifact in the whole system. Without it, ranking changes are opinions.

Staleness, and the enterprise's specific curse

The problem that has no web equivalent and deserves its own treatment.

An internal corpus accumulates documents that are outdated and still look authoritative. Five pages describe the deployment process; four are historical and none says so. Nothing about the text reveals which is current.

Serving a confidently formatted, professionally written, three-year-old process document is worse than serving nothing, because the reader has no reason to doubt it.

The available signals, roughly in order of reliability:

Explicit metadata — a review date, an owner, a lifecycle status. Best by far, and it usually does not exist, which makes creating it a higher-value intervention than ranking work.

Last modified, which is weak on its own — a typo fix updates the timestamp on a document whose content is years old.

Reference decay. How recently anything else linked to or mentioned it. A document nothing has referenced in two years is probably not current.

Contradiction with newer material. Expensive to compute and the strongest signal there is: two documents describing the same process differently, one recent, one not.

Access pattern. A document nobody has opened in a year, in an area people work in daily.

The product answer matters as much as the ranking one: show the date and the owner alongside every result, and surface an explicit warning when a retrieved document is old or conflicts with a newer one. Letting the reader judge is cheap and it is the honest response to a signal you cannot fully resolve.

Next: answering rather than finding.