Exploration as an Investment

In one line: exploration costs engagement today and buys information that improves every recommendation afterwards, which makes the question how much to spend rather than whether to do it.

The formal shape

Each item is an arm with an unknown payoff. You can pull the arm you currently believe is best — exploitation — or one you are uncertain about, which is worth less in expectation and tells you more.

The classic algorithms differ in how they value uncertainty:

Epsilon-greedy. With small probability, pick at random. Trivial to implement, and it wastes most of its budget on arms already known to be bad.

Upper confidence bound. Score each arm by its estimated value plus a term that grows with uncertainty, then take the maximum. Exploration concentrates where you are most unsure, which is where information is cheapest.

Thompson sampling. Keep a posterior over each arm's value, draw a sample from each, play the argmax. Exploration falls out of the uncertainty automatically, it handles delayed feedback gracefully, and it is what most production systems reach for.

Why the textbook version does not transfer

Four gaps between the formalism and a real recommender, and naming them is the depth signal.

The arms are contextual. An item is not good or bad in the abstract — it is good for some users. You are not learning one payoff per item but a payoff function, which is why contextual bandits rather than plain ones.

There are millions of arms, most of them new. Independent per-arm posteriors are hopeless. Uncertainty has to be shared through content features, so an unseen item inherits a prior from things like it.

Feedback is delayed and partial. A click arrives in seconds; watch time in minutes; whether it was a good recommendation, if you use surveys, in days. Thompson sampling tolerates this better than UCB, which wants its estimates updated promptly.

You show a slate, not an arm. Ten items at once, in positions with wildly different examination rates. The feedback is attributable to the slate as much as to any item, which breaks the clean per-arm accounting.

The design that survives contact

Most production systems do not implement a pure bandit over the catalogue. They do something coarser and more controllable.

Reserve a fraction of slots — one or two in ten — for items the ranking would not have chosen. Fill those slots by sampling from an eligible pool, weighted toward high uncertainty rather than uniformly. Log every exploration impression with a flag, so the resulting data is identifiable and can be weighted differently in training.

The flag is the part that gets forgotten and it is what makes the whole thing worth doing. Exploration data is the only unbiased sample you have. If it is indistinguishable from organic impressions in the logs, you have paid for it and cannot use it as what it is.

Centralising it

A platform with several recommenders — a feed, a search page, a notification system — that each explore independently pays for the same information several times over and gets an inconsistent user experience.

Spotify's published work describes centralising exploration and propagating what it learns back to the individual decentralised recommenders. The argument generalises: exploration is expensive and the information it produces is shared, so it is platform infrastructure rather than a per-surface concern.

What it costs, and where

The cost is not the exploration slot's lost engagement alone. Two more, and both are worth naming.

The variance cost. Exploration makes a user's experience less consistent. On a surface with strong intent — the item page for something specific — that is more damaging than on an open-ended feed. So the rate should differ by surface, and often by user: a brand-new user is the worst possible person to experiment on, because you have one chance to demonstrate value.

The measurement cost. Exploration traffic depresses your headline engagement metric, which makes every A/B test noisier and makes the team running exploration look worse. Reporting exploration and organic separately is the practical fix, and forgetting to do it is how exploration budgets get quietly cut.

What it buys

Three things, and only the first is usually mentioned.

Better estimates for uncertain items — the direct effect.

Unbiased evaluation data. Exploration impressions are a randomised sample, which makes them the only logged data you can use for counterfactual estimation without heavy assumptions. That is arguably worth more than the item estimates.

A working new-item pipeline. Exploration is the mechanism that makes cold start's exposure budget actually function, rather than a quota that fills with items nobody looks at.

The interview framing

Do not present exploration as a technique. Present it as a purchase:

Exploration spends engagement now to buy information that improves every future decision. I'd size it as a fixed one-to-two percent of impressions, weight it toward uncertainty rather than uniform randomness, vary it by surface — lower where intent is strong — and flag every exploration impression so the data is usable as the unbiased sample it is.

Next: the policy layer, and the constraints a loss function cannot express.