Requirements

Why this matters: the requirements list here is unusual because what it omits is more interesting than what it contains. Reading it critically is the exercise.

The backend design must address the following functional and non-functional requirements.

Functional requirements

Requirement	What the source says
Dialogue management	Interpret user input, maintain context, and generate relevant, human-like responses
Natural language understanding (NLU)	Identify intent and extract key entities from user input
Personalization	Adapt to user preferences and history to provide tailored interactions
Feedback	Improve accuracy and performance over time using user feedback

Only one of these is a request-time requirement — and that changes the architecture

Look at the timescales, because they are not the same and the design treats them differently.

Dialogue management, NLU, and personalization all happen during a request. They are on the critical path, they consume the latency budget, and they must complete before a response can be returned.

Feedback does not. A user pressing thumbs-down changes nothing about that response — it is already delivered. Feedback improves future responses, potentially weeks later after a retraining cycle.

That is why feedback in the detailed design goes through a pub-sub system and lands in a separate feedback database, rather than being handled inline. It is the same argument as billing, which we will meet in the same place: work that does not have to block the response should not block the response.

Naming this split is a cheap way to sound like you have built one of these. The request path and the improvement loop are different systems that happen to share a data model.

NLU as a separate requirement is a legacy of pre-LLM chatbots

"Identify intent and extract key entities" is the vocabulary of the intent-classification era — systems where a classifier mapped an utterance to one of N intents, slot-filled the entities, and dispatched to a handler.

A modern LLM does not work that way. There is no intent classification step; the model consumes tokens and emits tokens, and whatever "understanding" occurs is internal and not separately addressable.

The source nonetheless keeps a pre-processing NLU service as a distinct component. That is worth interrogating rather than accepting, and there are honest answers:

Embedding generation — converting the prompt to a vector is genuinely a separate step, and it is what makes the vector database and semantic cache possible.
Routing — classifying a prompt as simple or complex lets you send it to a smaller, cheaper model, which is a real production technique with real savings.
Safety pre-screening — catching prompt injection or disallowed requests before spending GPU time on them.
Normalization — language detection, truncation, template assembly.

So the component survives, but its justification has changed. If an interviewer asks why you have an NLU service in front of an LLM, "to extract intent" is a weak answer and "to embed, route, and pre-screen" is a strong one.

Non-functional requirements

Requirement	What the source says
Scalability	Handle increasing user traffic and workloads efficiently
Availability	Ensure consistent uptime for users
Reliability	Generate stable and predictable responses under varying conditions
Low latency	Minimize response times through efficient inference and caching
Privacy and security	Protect user data and ensure compliance with privacy standards

The relationships between them look like this:

"Stable and predictable responses" is a requirement the model cannot satisfy

Reliability here is defined as "generate stable and predictable responses under varying conditions." That is a much harder promise than it looks, because language model output is not deterministic by default.

Generation samples from a probability distribution over the next token. Two identical prompts can produce different answers — and that is deliberate, since it is what makes responses feel varied rather than canned. Even at temperature zero, floating-point non-associativity and changing batch composition can shift results between runs on the same hardware.

So "predictable" has to mean something weaker than "identical," and it is worth being precise about which:

Predictable quality — no answer is wildly worse than another. Achievable, and measured with an evaluation harness.
Predictable latency — a bounded time to first token. Achievable, and this is what admission control and batching policy actually defend.
Predictable content — the same prompt yields the same text. Not achievable, and mostly not wanted.

There is also a version of reliability specific to this domain that the list misses entirely: graceful degradation. When GPU capacity is exhausted, a web service queues or sheds load. An LLM service has a third option — fall back to a smaller model. Slightly worse answers beat no answers, and having that lever is one of the genuinely distinctive operational choices in AI systems.

Two requirements are missing, and they are the two that matter most

Cost is not listed. In every previous chapter that was defensible — servers are cheap enough that cost follows from scale. Here it does not. GPU inference is expensive enough that cost per request is a first-class design constraint, and it drives decisions no other requirement does: model size, quantization, batching policy, how aggressively you cache, whether cheap prompts are routed to a cheaper model.

A design that meets every requirement on this list and costs ten times what a competitor's costs has failed. Cost belongs in the non-functional list, and putting it there is a strong interview move.

Safety and factual accuracy are not listed either. The detailed design includes a content moderation system, so safety is present in the architecture but absent from the requirements — a real inconsistency. And accuracy, as the previous lesson argued, is not addressed anywhere at all.

This is a sharper version of a pattern the Google Maps chapter surfaced. There, accuracy was listed but had no architectural answer. Here, the equivalent requirement is not even listed. Systems tend to under-specify exactly the properties they cannot engineer, and noticing the gap is more valuable than reciting the list.

Privacy has a shape here it does not have elsewhere

"Protect user data" normally means encryption, access control, and retention policy — all of which apply.

But conversational AI adds two problems with no analogue in earlier chapters:

Prompts are unusually sensitive. People paste code, medical questions, legal disputes, and unreleased business plans into these systems. The content is often more revealing than the metadata any previous design stored.

Training on user data leaks it. If conversations feed back into training — which the feedback requirement explicitly contemplates — a model can memorize and later reproduce content from them. That is a privacy failure with no access-control fix, because the leak happens through the weights rather than through the database.

The mitigations are real but partial: opt-out controls, scrubbing personally identifiable information before it enters a training set, retention limits, and separating the serving path from the training corpus. Worth naming, because it is the kind of concern that only exists in this class of system.

Key takeaway

Four functional requirements — dialogue management, NLU, personalization, feedback — of which only feedback is off the critical path, which is why it runs asynchronously. Five non-functional ones, of which reliability's "predictable responses" cannot mean deterministic because generation samples. And two absent requirements matter more than any present one: cost, which uniquely drives model and batching decisions here, and factual accuracy, which has no architectural answer. NLU survives as a component but its justification has changed — embedding, routing, and pre-screening rather than intent classification.

Interview signal by level

Level	What a strong answer sounds like
L4	Recites the list: "dialogue management, NLU, personalization, feedback — and scalability, availability, reliability, low latency, privacy."
L5	Separates timescales: "three functional requirements are request-time and one — feedback — is an offline improvement loop, which is why it's asynchronous and behind pub-sub rather than inline."
Staff+	Adds what's missing: "I'd add cost as a non-functional requirement — GPU inference is expensive enough that cost per request drives model size, quantization, and batching policy in a way no other requirement does. And I'd flag that factual accuracy isn't listed anywhere, though moderation is in the architecture — so safety is designed for but not specified. I'd also be careful with 'predictable responses': generation samples, so predictable has to mean predictable quality and latency, not identical text. The distinctive lever here is falling back to a smaller model under load rather than shedding it."

Next: the resource estimation, and the arithmetic mistake that reveals what the system actually costs.

Requirements

Functional requirements

Non-functional requirements

Interview signal by level

Enjoying the preview?