What the Interviewer Is Scoring

In one line: the round has a shape, and knowing it lets you spend your attention on the two or three decisions that actually carry signal.

The spine

Every problem chapter in this course follows the same seven steps. It is not a script to recite — it is a checklist that stops you leaving a hole the interviewer will find.

Two notes on the order. Metrics come second, before any architecture, because everything downstream is justified against them. And failure comes last but is never optional — an answer that ends at "we deploy the model" has skipped the half of the lifecycle where these systems actually break.

Spending the time

Roughly, for a 45-minute round:

The deep dive is where the score is decided, so get through the earlier steps briskly and deliberately. Being crisp about scope is what buys you the time to be thorough about the hard part — and the interviewer will usually tell you which part that is, by what they ask about first.

What separates levels

The same question is scored differently depending on the bar, and the difference is rarely knowledge of an extra technique.

The Staff row has a theme worth extracting: it is about second-order effects. The system changes the data that trains its successor. The metric that measures it becomes a target that distorts it. The users it serves badly leave and stop appearing in the logs. A stronger model makes a bad proxy worse.

Reasoning at that level is the clearest separator in this round, and every one of those four sentences came from an earlier lesson in this chapter.

Reversibility is the underrated one

Saying which decisions can be walked back is cheap and it lands, because it is how senior engineers actually think about risk.

A model choice is reversible — retrain and redeploy. A metric definition is nearly irreversible once dashboards, teams and incentives are built on it. A schema for logged events is expensive to change because history cannot be back-filled. Choosing not to log propensities means counterfactual evaluation is unavailable to you forever.

Flagging the irreversible ones and spending more care there is a Staff behaviour that costs one sentence.

The five failure modes

Designing before scoping. Drawing boxes before establishing what the system is for. Everything after is unfalsifiable, because no choice can be justified against a goal nobody stated. This is the most common way a strong engineer gives a weak answer.

Treating the model as the system. Thirty minutes on architecture choices and five on everything else. The model is one box out of eight; the rest carries most of the score.

Presenting offline numbers as impact. Claiming a production win from a held-out score. It signals the candidate has not been through the loop where the offline win evaporates.

Stopping at deployment. No drift, no feedback loop, no retraining trigger, no monitoring. These systems degrade silently by default, so an answer with no operating story has never operated one.

Answering with vocabulary instead of reasoning. Naming two-tower retrieval, a feature store and a vector database without deriving why any of them are needed. Interviewers probe exactly here, and the follow-up question — "why two stages?" — has no answer if the pattern was recalled rather than derived.

Where the rest of the course goes

The building blocks module covers the reusable machinery these problems assemble from — vector search, feature stores, two-stage ranking, inference serving, retrieval pipelines, evaluation and guardrails. The problem modules then work through the questions themselves, from recommendation and fraud through to assistants and agents, each following the spine above.

Next: the walkthrough, where the spine gets run live against a deliberately vague brief.