Why LLM Systems Are Different
In one line: the architecture here looks like a normal request-response service with one unusual component. The unusual component changes the cost model, the latency model, and the failure model all at once.
The problem with rule-based bots
Traditional rule-based chatbots break down when user input falls outside predefined scripts, often returning fallback or nonspecific responses. This degrades the user experience and increases reliance on human escalation, which raises operational costs.
The economics of a support bot are entirely about deflection rate
It is worth being explicit about what this system is for, because it determines every trade-off in the chapter.
A support bot exists to resolve queries without a human. Every conversation it handles is a human interaction avoided; every conversation it fumbles is a human interaction plus a frustrated customer.
Rule-based bot: handles the scripted 30%, punts the rest
-> and the punts arrive at agents pre-annoyed
LLM bot: handles a much wider distribution
-> but can be confidently WRONG in ways a script cannot
That asymmetry is the whole design problem. A rule-based bot's failure is visible and safe — "I didn't understand that, let me connect you to an agent." An LLM's failure is invisible and confident — a fluent, well-formatted, entirely fabricated answer about your return policy.
Replacing a system that fails loudly with one that fails silently is a trade, not an upgrade, and most of this chapter's components — RAG, moderation, confidence scoring, escalation, feedback — exist to convert silent failures back into visible ones.
What the LLM adds, and what RAG adds on top
An LLM-powered bot... interprets intent, maintains conversational context across multiple exchanges, and generates natural responses. Modern production architectures go further by combining LLMs with retrieval-augmented generation (RAG), which grounds every response in company-specific knowledge bases and real-time data rather than relying on the model's static training data. This significantly reduces hallucination rates.
Parametric memory is stale by construction, which is the entire argument for RAG
The source's own aside states it precisely:
"Parametric memory refers to the knowledge baked into an LLM's weights during training. It becomes stale the moment source documents are updated, which is why retrieval-augmented approaches are essential."
That is the correct framing and it is worth extending, because it explains why RAG is not optional for this use case.
A model's weights are a snapshot. Whatever it knew at training time is what it knows, and updating that knowledge means retraining — weeks and enormous cost, for a return-policy change.
Support content changes constantly. Prices, policies, product specs, known issues, shipping times. The half-life of a support knowledge base is measured in days.
WITHOUT RAG: knowledge lives in WEIGHTS -> updating costs a training run WITH RAG: knowledge lives in a VECTOR DB -> updating costs a re-embed
So RAG is not primarily an accuracy technique. It is a decoupling technique — it separates what the system knows from what the model learned, so the two can change on completely different schedules.
That is the same instinct as every configuration-versus-code separation in this module: the thing that changes weekly should not live inside the thing that changes yearly.
'Significantly reduces hallucination' is the honest claim, and the chapter later treats it as elimination
Note the design's careful wording here — reduces, not eliminates — and then compare with the compliance table, which claims RAG "prevents hallucination."
The gap matters, because RAG introduces its own failure modes that Lesson 5 covers in full:
RETRIEVAL FAILURE: the right document isn't retrieved
-> the model answers from parametric memory ANYWAY,
just as confidently, with no signal that it did
IGNORED CONTEXT: the right document IS retrieved and the model
contradicts it -> grounding is a prompt instruction,
not an architectural guarantee
STALE INDEX: the document changed and the embedding didn't
Grounding is a prompt instruction, not an enforced constraint. Nothing in the architecture prevents the model from generating a sentence unsupported by the retrieved chunks — which is why production systems add citations and verification, and why the design's moderation layer is the last line rather than the only one.
RAG changes hallucination from a certainty to a probability, and designing as though it changed it to zero is the most common error in LLM system design.
The requirements
| Functional requirement | What the source specifies |
|---|---|
| Dialogue management | Multi-turn history, so 'what about the other item?' resolves within the session |
| Natural language understanding | Interpret intent, extract entities (order IDs, product names), disambiguate using context |
| Response generation | Fetch from the knowledge base / vector DB and generate grounded answers instead of hallucinating |
| Function calling | Invoke backend APIs when intent maps to an action — order status, initiate a refund |
| Human escalation | Detect low confidence and transfer while preserving full context |
| Feedback collection | User ratings, as a signal for improving quality |
Non-functional: scalability (millions concurrent, holiday peaks) · low latency (2–3 seconds) · availability (99.9%) · reliability (degrade gracefully) · cost efficiency (tiered models, caching, batching) · privacy and security (encryption, GDPR, access controls).
Function calling is listed as one requirement among six, and it changes the threat model entirely
Read that fourth row again: "invoke backend APIs when the user's intent maps to an actionable operation, such as checking order status or initiating a refund."
Every other requirement is about producing text. This one is about taking action — moving money, changing account state, creating tickets.
That single difference means the system is no longer just capable of saying something wrong. It is capable of doing something wrong, on the basis of a natural-language input from an untrusted party, interpreted by a model that can be steered by that input.
NO function calling: worst case is a wrong ANSWER -> embarrassing
WITH function calling: worst case is a wrong ACTION -> a refund issued
to someone who asked nicely
The design's answer is a content moderation service that "scans LLM outputs" — which is a control on text, placed after generation, and does not touch the action path at all.
Once an LLM can call functions, prompt injection stops being a content problem and becomes an authorization problem. Lesson 7 works through it; it is the most serious gap in the chapter.
Two non-functional requirements are in direct tension, and the design's headline optimization is the resolution
Low latency (2–3 seconds) and cost efficiency pull hard against each other, because in LLM serving they are governed by the same variable: which model you use.
Bigger model: better answers, HIGHER latency, HIGHER cost Smaller model: faster, cheaper, worse on hard queries
Most systems in this module resolved a latency/cost tension by caching or by precomputation. Neither works cleanly here — every query is different text, and answers cannot be precomputed for an open-ended input space.
So the design's answer is cost-aware routing: classify the query, and send easy ones to a cheap fast model and hard ones to an expensive slow one. That is genuinely the right shape, and Lesson 6 covers both it and the caching technique the chapter reaches for but does not name.
When latency and cost are governed by the same knob, the lever is not tuning the knob but routing to different settings of it.
What is genuinely new here
| Concern | In a conventional service | In an LLM service |
|---|---|---|
| Unit cost | A request costs fractions of a cent of CPU | A request costs real money — inference is the dominant line item |
| Latency source | I/O and network | Sequential token generation — proportional to output length (Lesson 3) |
| Capacity unit | RPS per CPU server | RPS per GPU — roughly 320× lower (Lesson 2) |
| Correctness | Deterministic — same input, same output | Probabilistic — same input, different output, sometimes wrong |
| Failure mode | An error, a timeout, a 500 | A fluent, confident, wrong answer |
| Knowledge freshness | Read the current database | Weights are a snapshot — hence RAG |
The row that reorganizes the architecture is capacity
Lesson 2 works the numbers, and the headline is worth having now: the chapter states that an app server handles 64,000 requests per second and a GPU inference server handles 200.
That is a 320× gap between two tiers of the same system.
Every architectural decision follows from it:
Why a cost-aware router? To keep queries OFF the expensive tier Why a cache? To keep queries OFF the expensive tier Why a lightweight model tier? To make the expensive tier cheaper Why batching? To raise the expensive tier's throughput
Four of the design's optimizations are the same optimization, viewed from different angles: reduce the number of tokens the big model has to generate.
When one tier of a system is hundreds of times weaker than the tier in front of it, the architecture becomes a filter — and every component before the GPU exists to reduce what reaches it.
Key takeaway
An LLM bot replaces a system that fails loudly with one that fails silently — a fluent, confident, fabricated answer where a script would have said "I don't understand" — and most of this chapter's components exist to convert those failures back into visible ones. RAG is primarily a decoupling technique, separating what the system knows from what the model learned so they can change on different schedules; it makes hallucination a probability rather than a certainty, since grounding is a prompt instruction and not an enforced constraint. Two things reorganize the architecture: function calling turns a wrong answer into a wrong action, which makes injection an authorization problem rather than a content one; and a 320× capacity gap between the app tier and the GPU tier turns the entire design into a filter whose job is to reduce what reaches the model.
Next: the estimation, and the number that actually constrains this system.