APIs and the Architecture
In one line: the component list is unusually complete for an LLM design. Reading it as a filter rather than as a pipeline is what makes the ordering make sense.
The APIs
sendMessage(session_id, user_id, message) -> orchestrates NLU, RAG, LLM parseQuery(text) -> intent + entities retrieveAndGenerate(query, session_id) -> vector search + grounded generation executeAction(action, parameters) -> function calling into backend APIs escalateToHuman(session_id, reason) -> handoff with full context submitFeedback(session_id, message_id, rating, comment?)
sendMessage carrying session_id is what makes the bot conversational rather than transactional
The first parameter is the one doing the work. Without it, every message is an isolated query and "what about the other item?" is unanswerable.
With it, the system can do the thing the first functional requirement asks for:
Turn 1: "Where's my order 4471?" -> resolves order 4471 Turn 2: "What about the other item?" -> needs turn 1 to mean anything
That makes the session the unit of state in an otherwise stateless architecture — the app servers, the router, the retrieval service, and the LLM servers are all stateless, and the conversation is the only thing that persists across requests.
Which is why Redis holds it rather than the primary database: it is small, hot, read on every single turn, and has a natural expiry. Put the one piece of per-request state in the fastest store you have, because it is on the critical path of every request.
The cost is covered in Lesson 8 and worth flagging now: conversation history is re-sent to the model on every turn, so a long session makes each request larger and slower — and Lesson 3 established that prompt length is what drives time-to-first-token.
parseQuery and retrieveAndGenerate are internal, and listing them as APIs blurs a boundary worth keeping
Look at what each API is for:
EXTERNAL (a client calls these): sendMessage() submitFeedback() INTERNAL (a service calls these, as a step inside sendMessage): parseQuery() retrieveAndGenerate() executeAction() escalateToHuman()
The source describes parseQuery as "used by the NLU service" and retrieveAndGenerate as a RAG step, so they are pipeline stages rather than entry points.
Two of them should be explicitly not client-callable, and this is not pedantry:
executeAction(action, parameters) issues refunds. If a client can call it directly, the LLM is irrelevant to the security question — you have exposed an unauthenticated action endpoint. Lesson 7 covers the deeper version of this problem.
escalateToHuman(session_id, reason) consumes a scarce, expensive resource: an agent's time. A client-callable escalation endpoint is a denial-of-service on your support organization.
Distinguish orchestration steps from the public surface, and be explicit about which internal capabilities must never be reachable from outside — particularly ones that move money or consume human attention.
submitFeedback carrying message_id rather than session_id is a small, correct detail
submitFeedback(session_id, message_id, rating, comment?)
Feedback attaches to a specific response, not to the conversation. That matters because a session contains many turns, and "this conversation was bad" is not actionable while "this answer was wrong" is.
With message_id you can join back to everything that produced that answer:
message_id -> which model tier served it (Lesson 6)
-> which chunks were retrieved (Lesson 5)
-> what the confidence score was (Lesson 8)
-> whether it was a cache hit
That join is what turns feedback from a satisfaction metric into a debugging signal — you can ask "do thumbs-down responses correlate with low retrieval scores?" and get an answer that tells you whether the problem is retrieval or generation.
Attach feedback to the smallest unit you can trace, and record the provenance of each response so the join is possible. Lesson 8 covers what the feedback can and cannot tell you.
The architecture
| Component | Role | What it really does |
|---|---|---|
| Entry layer + app servers | Auth, rate limiting, traffic distribution, business logic | Rate limiting is a cost control here, not just abuse prevention |
| Pre-processing (NLU) | Classify intent, extract entities | Triage — decides what happens next |
| Cost-aware router | Simple → lightweight model; complex → full LLM | Keeps queries off the GPU tier (Lesson 6) |
| RAG pipeline | Embed, search, rank, inject top-k as grounding | Decouples knowledge from weights (Lesson 5) |
| LLM core servers | Generation with function calling | The bottleneck — and the action surface (Lesson 7) |
| Vector database | Chunked, embedded docs, FAQs, policies | Updated by a knowledge ingestion pipeline |
| Redis | Per-session conversation history | The system's only per-request state |
| Content moderation | Scan outputs for policy violations and PII | ⚠️ Output-only — nothing scans inputs (Lesson 7) |
| Human escalation | Confidence scoring → live agent with context | The safety valve — and confidence is undefined (Lesson 8) |
| Monitoring | Accuracy, latency, CSAT, cost per query | Cost per query as a first-class metric is unusual and right |
| MongoDB | Logs, profiles, tickets, escalations | Audit and analytics — and a privacy liability (Lesson 2) |
Read the request path as a filter, and the ordering explains itself
The stages look like an arbitrary pipeline until you notice what they have in common: everything before the LLM exists to reduce what reaches it.
Rate limiter -> reject excess requests entirely
Cache -> answer without generating at all
NLU triage -> classify so routing is possible
Cost-aware router-> send the easy ones to a cheap model
RAG top-k -> bound how much context the model must process
============================================
LLM CORE <- 320x weaker than the tier in front
Each stage is cheap and each removes load from the one component that is expensive. That is the same shape as the typeahead chapter, where debouncing, an input threshold, and local caching removed 60% of requests before they were sent — the cheapest request is the one that never reaches the expensive tier.
The ordering follows from cost: the cheapest filter goes first. Rate limiting costs nothing; a cache lookup costs a millisecond; NLU classification costs a small model call; retrieval costs a vector search. Nothing expensive is spent deciding whether to spend something expensive.
When one component dominates cost, design the path to it as a sequence of increasingly expensive filters.
'Cost per query' as a monitored metric is unusual and it belongs here
Buried in the monitoring component: it tracks "LLM response accuracy, latency, user satisfaction scores, and cost per query."
No other system in this module monitors unit cost as a first-class operational metric, and the reason it belongs here is Lesson 2's arithmetic: at 250 million requests a day, a tenth of a cent per query is $91 million a year.
That makes cost an SLO, not a budget line:
Cost per query drifts up -> the router is misclassifying and sending easy queries to the big model -> or the cache hit rate has fallen -> or prompts have grown (more chunks, longer history) -> or a model change altered output length
Every one of those is a regression that no other metric catches. Accuracy stays fine, latency stays fine, and the bill doubles.
When unit costs are material, cost per request is a health metric — and monitoring it is what makes the cost-aware router's effectiveness observable rather than assumed.
The workflow puts escalation after moderation, which is one step too late
Trace the design's own numbered workflow:
7. LLM generates a response (possibly calling functions) 8. Content moderation scans the response 9. IF confidence is below threshold -> escalate to a human 10. The validated response is returned
The confidence check happens at step 9, after generation and after moderation. But the point of escalation is that the system cannot answer reliably — and if that is true, generating the answer was wasted work on the most expensive tier in the system.
More importantly, some of the signal is available before generation:
Retrieval returned nothing above the relevance threshold -> escalate NOW NLU classified the intent as one requiring a human -> escalate NOW The user has explicitly asked for a person -> escalate NOW The user has expressed frustration -> escalate NOW
None of those requires an LLM response to detect, and all of them are stronger evidence than a post-hoc confidence score.
Escalate as early as the evidence allows — it saves the generation cost, and more importantly it saves the user from reading a hedged non-answer before being handed off. Lesson 8 covers what confidence actually is and why pre-generation signals are more reliable than post-generation ones.
Key takeaway
sendMessage carrying a session_id makes the session the only per-request state in an otherwise stateless architecture, which is why it lives in Redis — and why prompt length grows with conversation length. Four of the six APIs are orchestration steps rather than entry points, and two of them — issuing refunds and consuming agent time — must be explicitly unreachable from outside. The architecture's ordering makes sense once you read the request path as a sequence of increasingly expensive filters in front of a tier 320× weaker than the one before it, with the cheapest filter first. Cost per query as a monitored metric is unusual and correct — at this volume it is an SLO, and it is the only metric that catches router misclassification, cache-rate decay, or prompt growth. And escalation is checked one step too late: the strongest signals are available before generation.
Next: the RAG pipeline, and what it does and does not fix.