Interleaving

In one line: instead of showing ranker A to half your users and ranker B to the other half, mix both into one list and see which one's results get clicked.

Why an A/B test is a blunt instrument here

An A/B test compares two populations. The signal you want — is ranking B better than ranking A — is buried under everything that differs between two sets of people: different queries, different intents, different sessions, different days.

Between-subject variance dominates, so you need a lot of traffic to detect a small ranking improvement. On a search team shipping ranking changes weekly, that is the bottleneck.

Interleaving removes the variance by making the comparison within a single impression. The same user, on the same query, at the same moment, sees results from both rankers and picks.

Team-draft interleaving

The standard construction, and it is named after picking teams in a playground.

Two rankers produce two lists. Treat them as captains taking turns, in randomised order, each picking their next-highest result not already taken. The merged list has results from both, and every result is attributed to whichever ranker picked it.

Then count clicks by attribution. If A's picks get clicked more often than B's across many impressions, A wins.

A: [a1, a2, a3, a4]      coin flip -> B picks first
B: [b1, b2, b3, b4]

merged: b1(B) a1(A) b2(B) a2(A) ...
        attribution recorded per position

Two details make it sound. The randomised pick order per impression prevents either ranker from systematically getting the top slot, which would otherwise decide the outcome by position bias alone. And a result both rankers would have picked goes to whoever drafted it, so agreement contributes no signal — which is correct, since agreement tells you nothing about which is better.

The sensitivity gain

This is the number that justifies the machinery: interleaving reliably detects ranking differences with substantially less traffic than an A/B test. Published large-scale comparisons of interleaved evaluation consistently find A/B testing the least sensitive of the methods studied, with interleaving variants requiring materially fewer impressions to reach a reliable outcome.

The practical consequence is a change in how a team works. A comparison that needs two weeks as an A/B test may resolve in a day or two interleaved, which turns ranking iteration from a fortnightly cycle into a daily one.

Where it misleads

Three cases, and naming them is what separates knowing the technique from knowing when to use it.

It measures preference, not impact. Interleaving tells you which ranker's results get clicked more. It says nothing about revenue, session length, retention, or whether the user was satisfied. A ranker that wins an interleaving comparison can still lose on the business metric — so interleaving picks the candidate and an A/B test confirms it.

It only works when the lists overlap in kind. If A and B return substantially different types of result — one returns products and the other returns categories — the merged list is incoherent, and the user's clicks reflect the strangeness of the page as much as the quality of either ranker. Interleaving compares two orderings of similar things; it does not compare two different experiences.

Whole-page changes break it. Changing layout, snippets, images or the number of results cannot be interleaved, because those are properties of the page and not of individual results. Only the ordering can be mixed.

The bias it inherits

Interleaving cancels between-user variance and does not cancel position bias.

Users click higher results regardless of quality, so if the interleaving policy hands one ranker better positions on average — through an implementation asymmetry, or through unequal contributions when one list is shorter — the attribution is biased. Team-draft's randomised pick order handles the common case; published work on debiased and balanced interleaving addresses the remaining asymmetries, and the reason that literature exists is that the naive implementations do leak.

Practical version: randomise the pick order per impression, verify empirically that each ranker receives statistically equal position distributions, and treat a systematic position imbalance as a bug in the interleaving rather than a result.

Where it fits

The three instruments, in the order you use them:

Judgements gate the change offline. Interleaving picks among the candidates that survive. The A/B test confirms the winner against the metric that actually matters, and it is the only one of the three that can measure a whole-page or business effect.

A team using only the third iterates slowly. A team using only the second optimises clicks. Using all three is what the structure is for.

Next: indexing, freshness, and serving it inside the budget.