Relevance Judgements

In one line: a human can look at a query and a document and say whether they match, which gives search a benchmark that does not move when your system does.

Why this is a structural advantage

A recommender's offline evaluation is scored on data its own previous policy generated, so it rewards agreement with the system being replaced. Search escapes that.

Given waterproof hiking boots and a product page, a trained rater grades the match without knowing what your ranker did. The label is independent of your system, so an improvement is measurable as an improvement rather than as agreement.

That single property is why search teams can iterate offline with some confidence, and it is worth naming explicitly when the comparison to recommendation comes up.

Graded, not binary

Relevance is not a yes or no. The standard is a graded scale, typically five levels — perfect, excellent, good, fair, bad — because the distinction between an ideal answer and an acceptable one is exactly what ranking is about.

Grades feed NDCG, which weights by position and by gain, so a perfect result at rank one is worth much more than the same result at rank five. Binary labels throw away the information that makes the metric meaningful.

Building the judged set

Four decisions, and each one has a failure attached.

Which queries. Stratified across head, torso and tail, not sampled by traffic — a traffic-weighted sample is nearly all head, and every tail regression becomes invisible. Include known-hard queries and ones that have caused incidents.

Which documents. You cannot judge the whole corpus per query. The standard approach is pooling: take the top results from several different systems, judge the union. It is efficient and it has a known bias — a document no system in the pool retrieved is never judged, so it is implicitly treated as irrelevant. A genuinely novel retrieval method can look worse than it is because its best finds were never in the pool.

Who judges. Trained raters with written guidelines. The guidelines are the real artifact: what counts as perfect, how to treat partial matches, how to handle ambiguous queries. Without them, you are measuring rater opinion rather than relevance, and it drifts.

How many per pair. Multiple raters on a sample, so you can measure agreement. If your raters do not agree with each other, no model can be expected to agree with them either — inter-rater agreement is the ceiling on any metric computed from these labels.

The four ways it stops being true

A judged set is a snapshot, and every one of these erodes it silently.

The corpus moves. New documents appear that were never judged, and judged documents are deleted or change. In e-commerce this is fast — a judged product goes out of stock and the label is now wrong by that product's own definition of relevance.

The intent moves. A query's meaning changes. swift meant one thing before a programming language existed. Seasonal queries mean different things in June and December.

The guidelines drift. Raters are retrained, the guidelines are edited, and labels from eighteen months ago encode a different standard. Comparing a metric across that boundary compares two definitions.

The pool bias hardens. Every judgement round pools from your current systems, so the judged set increasingly reflects what your systems retrieve — which reintroduces exactly the self-confirming property that judgements were supposed to avoid.

That last box is the depth point. Judgements avoid the closed loop only if the pool contains results your current systems would not return — which means deliberately pooling from a random or exploratory retrieval, not only from your production variants.

What judgements do not tell you

The limitation to state before an interviewer does.

Judged relevance is not satisfaction. A rater can confirm a result matches the query and cannot tell you whether the user was happy, whether the page loaded, whether the price was acceptable, or whether they bought anything.

Raters are not your users. They follow a rubric; your users have context, history and intent the rater cannot see. On personalised or intent-ambiguous queries a rater's judgement is a guess about a typical user.

It says nothing about the whole-page experience. NDCG over ten results ignores the layout, the facets, the snippets and the ads. A page can score perfectly and be unusable.

So judgements are one leg. The complete evaluation story is judgements for a stable offline benchmark, interleaving for a sensitive online comparison, and A/B tests for the business decision — three instruments measuring three different things.

The cost, and where to spend it

Judgements are expensive and the budget is finite, so spend it where it discriminates.

Queries where systems already agree waste budget — every candidate returns the same thing and the labels change no decision. The high-value judgements are on queries where your candidate systems disagree, because those are the ones where the label decides. Pooling naturally surfaces these: a document retrieved by one system and not another is exactly the case worth adjudicating.

Refresh continuously rather than in large periodic rounds, so the set ages evenly and no single re-labelling event creates a discontinuity in your metric history.

Next: the online instrument that needs far less traffic than an A/B test.