Evaluation
In one line: these tables are specific enough to check, and three of the rows claim more than the architecture delivers.
The property that makes this hard: the same input can produce different output, so a test suite that asserts exact strings fails constantly and teaches the team to ignore it. Assert on properties — is it grounded, does it cite, does it refuse when it should — not on text.
Functional requirements
| Requirement | Stated mechanism | Assessment |
|---|---|---|
| Grounded responses | RAG retrieves current documentation from the vector database, preventing hallucination | 🔴 Reduces, does not prevent — retrieval failure is invisible (Lesson 5) |
| Multi-turn conversation | Redis maintains session history, injected into each prompt | ⚠️ Correct — and unbounded; prompt grows every turn (Lesson 8) |
| Function calls | The LLM invokes external APIs for real-time data | 🔴 No authorization on the action path (Lesson 7) |
| Human escalation | Confidence scoring triggers routing via pub-sub with full context | ⚠️ Context preservation is right; confidence is never defined (Lesson 8) |
| Feedback collection | Structured and unstructured feedback stored in MongoDB | ⚠️ Collected correctly; measures satisfaction, not accuracy (Lesson 8) |
These rows name components, which is the test most compliance tables in this module fail
Worth crediting: every row points at a specific thing in the architecture — the vector database, Redis, pub-sub, MongoDB. Compare with the tables that said "redundancy is ensured" or "adhere to industry security standards."
Because they are specific, the gaps are checkable rather than rhetorical — which is what produces the three warnings above.
A compliance row is only useful if the design can be checked against it. These can be, and three of the five do not survive the check.
'Preventing hallucination' is the claim that most misleads, because it changes what you build
The introduction said RAG "significantly reduces hallucination rates." The compliance table says it prevents hallucination.
That is not a wording quibble, because the two beliefs produce different systems:
IF RAG PREVENTS hallucination: -> no citations needed (nothing to verify) -> no grounding metric needed (always grounded) -> no retrieval-quality monitoring (retrieval is assumed to work) -> confidence scoring can rely on the model IF RAG REDUCES hallucination: -> citations, so ungrounded claims are detectable -> a grounding rate you can alert on -> retrieval score as the primary confidence signal -> monitoring for retrieval failure specifically
The design has none of the second column, which is consistent with believing the first.
And Lesson 5's failure mode is the one that punishes it: when retrieval returns nothing relevant, the model answers from parametric memory in exactly the same confident register, with no signal distinguishing the two. The stronger claim removes the reason to build the detection.
A design that assumes a mechanism is perfect will not contain the instruments to notice when it isn't.
Non-functional requirements
| Requirement | Stated mechanism | Assessment |
|---|---|---|
| Scalability | Load balancing plus horizontal scaling of stateless services (LLM servers, RAG, vector DB) | ⚠️ True of every tier except the one that matters — GPUs are capacity-bound, not stateless-bound |
| Low latency | Cost-aware routing, Redis caching, efficient retrieval → 2–3 seconds | 🔴 Unreachable without streaming, which is never mentioned (Lesson 3) |
| Availability | Redundant gateway, load balancer, distributed servers, monitoring → 99.9% | ⚠️ Generic — and no degraded mode is defined |
| Reliability | RAG grounding ensures consistent responses; graceful degradation via routing and escalation | 🔴 LLM output is not consistent — same input, different output |
| Cost efficiency | Tiered model selection, caching, batched inference | ✅ The right three mechanisms — and cost is never computed (Lesson 6) |
| Privacy and security | Gateway auth, encryption, content moderation prevents PII leakage | 🔴 No input moderation, no action authorization (Lesson 7) |
'Horizontal scaling of stateless services' is true and does not describe the bottleneck
The scalability row lists the LLM servers among the stateless services that scale horizontally. Statelessness is not what constrains them.
App servers: stateless AND cheap -> add instances freely
GPU servers: stateless AND SCARCE
-> 320x lower throughput per node (Lesson 2)
-> a large model needs ~25 accelerators just to hold its weights
-> accelerator supply and cost are the real limits
Being stateless makes a tier easy to scale; it does not make it cheap to scale. For the GPU tier, adding capacity is a procurement and budget question, not a deployment question.
Which is why the design's real scalability answer is not in this row at all — it is the cost-aware router, the cache, and batching, all of which reduce demand on the constrained tier rather than expanding it. When a tier cannot be scaled cheaply, the scalability strategy is demand reduction, and the design does this well while describing it under "cost efficiency" instead.
'RAG grounding ensures consistent responses' — LLM output is not consistent by construction
The reliability row claims consistency, and generation is stochastic: the same query, with the same retrieved context, can produce materially different answers on two invocations.
Two identical queries, same retrieved chunks:
Response A: "Returns are accepted within 30 days of delivery."
Response B: "You have 30 days from when your order arrives to return it.
Note that sale items are final."
Both may be correct. They are not the same answer, and the second volunteers a condition the first omitted — which is the version of inconsistency that generates complaints.
RAG constrains the content and does nothing about the variance. What actually reduces it:
Temperature at or near zero for factual support answers. There is no value in creative variety when stating a return policy.
Semantic caching (Lesson 6), which makes repeated questions return the identical answer rather than a fresh sample.
Templated responses for the top intents — the highest-frequency queries do not need generation at all, which is also the cheapest possible answer.
Consistency in an LLM system is bought by not generating, through caching and templating, rather than by grounding. And it matters more than it sounds: two customers asking the same question and receiving different policy answers is a support failure regardless of whether either was wrong.
Availability names redundancy and never defines what 'degraded' looks like
99.9% uptime with redundant components is the generic answer. The interesting question for this system is what happens when the GPU tier is saturated or unavailable — because that is the tier that cannot be scaled quickly.
The design says the system should "degrade gracefully under heavy load rather than returning errors or nonsensical outputs" and never says how. The ladder is worth having:
1. Serve from the SEMANTIC CACHE — no inference at all 2. Route EVERYTHING to the lightweight model — degraded answers, still useful 3. Fall back to RETRIEVAL ONLY — show the relevant policy document, no generated summary. Accurate, less pleasant. 4. Fall back to a SCRIPTED bot for the top intents 5. Queue for HUMAN escalation 6. Only then: an error
Step 3 is the one people miss and it is strong: the retrieved documents are useful on their own. A customer shown the actual return policy is better served than one shown an error, and it costs no inference.
Define the degradation ladder explicitly, because "degrade gracefully" without a specification means the system will fail in whatever way the code happens to fail.
Note also that steps 1–4 all reduce load on the constrained tier, so the degradation ladder and the cost strategy are the same mechanisms at different thresholds.
What the evaluation omits
Five things a strong answer adds, each covered earlier:
Streaming. The 2–3 second budget is unreachable for a 1,000-token response; the metric must be time-to-first-token (Lesson 3).
Cost, computed. A stated requirement with no number; at this volume it is roughly $91M/year at a tenth of a cent per query (Lesson 6).
Prompt injection and action authorization. The model can issue refunds and nothing checks permissions independently of the prompt (Lesson 7).
What confidence is. Escalation depends on a threshold applied to an undefined quantity (Lesson 8).
Bounded conversation memory. History grows every turn, making long sessions slower and more expensive at once (Lesson 8).
Key takeaway
The compliance tables name components rather than qualities, which makes their gaps checkable — and three claims exceed their mechanisms. "Preventing hallucination" is the most consequential, because a design that assumes a mechanism is perfect will not contain the instruments to notice when it isn't: no citations, no grounding rate, no retrieval-failure monitoring. "Horizontal scaling of stateless services" is true and misses the bottleneck — being stateless makes a tier easy to scale, not cheap, so the real scalability strategy is demand reduction, which the design does well under a different heading. "RAG ensures consistent responses" is false by construction, since generation is stochastic; consistency is bought by not generating — low temperature, semantic caching, and templates for top intents. And availability names redundancy without a degradation ladder, whose strongest rung is the one people miss: serve the retrieved document itself.
Next: the interview walkthrough.