Interview Walkthrough: Framing an Ambiguous Brief Live

The prompt: "Our search isn't very good. Design something better."

That is the whole brief. It is vague on purpose, and how you handle the first five minutes decides most of the outcome.

Step 0 — Scope it

Do not draw anything yet. "Better" is undefined, and every architecture decision depends on which definition is meant.

Candidate: "Before I design, I want to pin down what 'not very good' means, because the fix differs completely. Are users not finding what they want, finding it too slowly, or finding it but not engaging with it? And what kind of search is this — product, document, media, code?"

Interviewer: "E-commerce product search. Users complain they can't find things they know we stock."

That is a lot of information. "Can't find things we stock" is a recall problem, not a ranking problem, and those have different fixes. Say so, because it is the first place you can demonstrate that you are listening rather than pattern-matching.

Candidate: "That points at retrieval rather than ranking — if the item never enters the candidate set, no amount of re-ranking recovers it. I'd want to confirm that before optimising the wrong stage. Two more: roughly how many products, and what latency is acceptable?"

Interviewer: "About 50 million SKUs. Search feels slow today; assume 200ms end to end is the target."

Assumptions to state and move on with: English-language queries, an existing keyword index, personalisation available but not required in v1, read-heavy traffic, and an eventually-consistent catalogue.

1. Metrics before architecture

Candidate: "My primary metric is successful-search rate — the fraction of searches that end in a click through to a product page, and ideally a purchase. I'd hold three guardrails: p99 latency, zero-result rate, and revenue per session."

Then name the proxy problem before being asked:

Candidate: "Click is a proxy and it's biased by position — the top result gets clicked partly because it's on top. So I'd measure at the session level rather than per result, and treat a reformulated query as a signal that the previous search failed."

Query reformulation as a failure signal is a strong, cheap observation. A user who searches, looks, and immediately searches again has told you the first attempt failed, with no annotation required.

2. Data

Candidate: "Labels come from the search log — query, results shown, position, clicks, add-to-cart, purchase. That's abundant and biased: I only ever observe judgements about items the current engine surfaced."

That sentence sets up the entire evaluation discussion later, so it is worth spending a line on now rather than defending it under pressure in fifteen minutes.

The features split cleanly, and saying which side each falls on shows you are thinking about the serving path rather than a notebook:

The last row matters commercially and is a good place to show product judgement: showing an out-of-stock item is a guardrail failure, and stock is exactly the feature you cannot precompute.

3. High-level design

Draw it in stages and narrate each upgrade rather than presenting the finished thing.

(a) What exists today — one keyword index, lexical matching only.

Candidate: "A pure lexical index fails on vocabulary mismatch. Someone searching 'laptop sleeve' doesn't match a product titled 'notebook case', and that's exactly the 'you stock it and I can't find it' complaint we started with."

(b) Add semantic retrieval alongside it, not instead of it.

Candidate: "Hybrid rather than pure vector, because lexical is still better at exact matches — model numbers, brand names, SKUs. A customer searching 'A2338' wants that exact part, and an embedding will happily return something semantically similar and useless."

That example is the reason hybrid retrieval is the production standard, and the concrete version lands far better than the abstract claim.

(c) Separate retrieval from ranking with explicit budgets.

Retrieval optimises recall over 50 million items in roughly 30ms. Ranking optimises ordering over a few hundred in roughly 50ms. The business layer is cheap and last, because a policy rule should override a model rather than compete with it.

(d) Precompute everything that does not depend on the query. Product embeddings are query-independent, so they are computed offline and indexed; only the query tower runs online.

4. Deep dives

Interviewer: "How do you know the semantic retrieval actually helped?"

This is where I'd expect offline results to overstate the win. My logs only contain clicks on items the lexical engine surfaced, so an offline metric rewards a model for reproducing lexical results — the exact items semantic retrieval is meant to add have no positive labels, because nobody was ever shown them. I'd use offline scoring only to reject clearly worse candidates, then run interleaving, which blends both rankings for the same user and detects a difference on far less traffic than an A/B test.

Interviewer: "The zero-result rate drops to nearly zero. Is that good?"

On its own, no — that's the metric behaving exactly as Goodhart predicts. Semantic retrieval always returns something, so zero-result rate can go to zero while the results get worse. I'd pair it with a relevance guardrail: a similarity floor below which we deliberately return nothing and say so. "No results" is a better experience than twenty irrelevant ones, because it tells the user to reformulate instead of teaching them the search is useless.

Interviewer: "Where does this system create a feedback loop?"

In the training data. Whatever the engine ranks highly gets clicked more, which produces more positive labels for those items, which trains the next model to rank them higher. Popular products get more popular and new products struggle to ever be surfaced. I'd counter it with position-debiased training — weighting a click by the inverse propensity of that position being examined — and a small exploration budget that deliberately surfaces uncertain items.

Interviewer: "A feature pipeline breaks at 2am. What happens?"

Nothing visible, which is the problem. If the popularity feature starts returning nulls, the model treats null as a value and the ranking quietly gets worse at full availability. So I'd monitor the score histogram and per-feature null rates — that catches it in minutes without needing any labels — and I'd make the model tolerate an explicitly absent feature rather than imputing a zero. The on-call action is to disable that feature and fall back, not to debug the model.

Interviewer: "What's your fallback?"

Lexical-only retrieval with popularity ranking. Much worse than the full system and enormously better than an outage, and critically it shares no infrastructure with the semantic path — so an ANN index problem can't take it down too. I'd exercise it on a small traffic slice continuously, because a fallback that hasn't run in six months is a hypothesis rather than a fallback.

Interviewer: "What breaks slowly?"

Embedding staleness. The catalogue changes daily but the embedding model is retrained rarely, so new products get embedded by an increasingly out-of-date model and the index drifts from the catalogue. Nothing errors. I'd monitor the age distribution of the index and the share of queries whose top results are stale — and I'd slice the quality metrics by segment, because an aggregate that looks flat can hide one category collapsing.

Now do it live