Cost-Aware Routing and Semantic Caching

In one line: cost is a stated non-functional requirement and is never quantified. Once you compute it, both of this lesson's mechanisms stop looking like optimizations and start looking like requirements.

The cost nobody computes

At 250 million requests a day, a tenth of a cent is ninety-one million dollars

The requirements say "LLM inference is expensive at scale, so the architecture must incorporate tiered model selection, response caching, and batched inference to control operational costs." Correct, and no number appears anywhere in the chapter.

Using the chapter's own volume:

250,000,000 requests/day

At $0.0005/request:  $125,000/day  =  $46M/year
At $0.001 /request:  $250,000/day  =  $91M/year
At $0.005 /request: $1,250,000/day = $456M/year

Cost is not a line item at this volume; it is the dominant operational constraint — larger than servers, storage, and bandwidth combined, by orders of magnitude. Lesson 2 showed bandwidth is under 20 MB/s and storage is 1.5 TB/day, both trivially cheap.

That reframes every optimization in this lesson. A 50% cache hit rate is not a performance improvement; it is $45 million a year. A router that keeps 70% of queries on a cheap tier is not tuning; it is the difference between a viable product and one that loses money on every conversation.

And it explains two design choices that look ordinary elsewhere:

The rate limiter is a cost control, not primarily an abuse control — an unthrottled client is an unbounded bill.

Cost per query is a monitored metric (Lesson 4), because it is the only signal that catches router drift, cache decay, or prompt growth. Accuracy stays fine, latency stays fine, and the bill doubles.

When unit cost is material, compute it before designing the optimizations — otherwise you cannot tell which ones matter.

Cost-aware routing

This component evaluates query complexity and routes simple, high-confidence requests to a lightweight, lower-cost model, and complex or ambiguous requests to a full-capability LLM. Without this layer, every FAQ lookup burns the same compute budget as a multi-step troubleshooting session.

The accuracy targets descend, and that inversion is the design's cleverest detail

Read the last column again: 95% for simple queries, 90% for moderate, 85% for complex. The system is held to a lower standard on the hardest questions.

That looks backwards and it is exactly right, because the tiers have different consequences of being wrong:

SIMPLE:   wrong answer -> the user acts on bad information, unnoticed
          -> nothing catches it. So the bar must be HIGH.

COMPLEX:  wrong answer -> a human is already in the loop
          -> the error is caught before it reaches the customer
          -> so a lower bar is affordable

Accuracy requirements should track the cost of an undetected error, not the difficulty of the question. Where there is a safety net, you can accept more errors; where there is none, you cannot.

It also explains why simple queries go to a fine-tuned small model rather than a general one. A small model fine-tuned on your FAQ corpus can beat a large general model on that narrow distribution — narrow and trained beats broad and general within the narrow domain, which is what makes the cheap tier viable at a 95% bar.

The router is a classifier, and its errors are asymmetric

Everything depends on correctly judging complexity before answering, which is itself a prediction — and it can be wrong in two directions with very different costs:

UNDER-ROUTE (hard query -> cheap model):
  -> a confident wrong answer from a model that couldn't handle it
  -> the user is harmed; nothing detects it
  -> EXPENSIVE in trust

OVER-ROUTE (easy query -> expensive model):
  -> a correct answer that cost more than it needed to
  -> EXPENSIVE in dollars only

These are not symmetric, so the threshold should not be centred. Bias toward over-routing: paying a few cents extra beats giving a customer wrong billing information.

The source's own practical tip is well judged and worth keeping:

"Start by logging query classifications for two weeks before activating routing decisions. This lets you validate the classifier's accuracy without risking degraded responses for real users."

That is shadow mode, and it is the same pattern as the canary deployment in that building block: run the new decision path, record what it would have done, compare against what actually happened, and only then let it act. Evaluate a decision system against reality before letting it make decisions.

One gap: the router has no fallback. If the cheap model produces a low-confidence answer, the design has no path to retry on the big model. A cascade — cheap tier first, escalate to the expensive tier when confidence is low — captures most of the savings while bounding the under-routing risk, and it is strictly better than a one-shot classification.

Caching

The cache layer stores frequent query-response pairs to reduce redundant LLM inference calls, directly supporting cost-aware routing.

Exact-match caching on natural language has a terrible hit rate — and the fix is already in this design

The intent is right and the mechanism implied is wrong. Support queries are free text, and users express one intent in unlimited ways:

"What is your return policy?"
"whats ur return policy"
"How do I return something?"
"can i send this back"
"I want to return an item I bought last week"
"return policy?"

Six queries, one intent, six different cache keys. Exact-match caching hits on none of them after the first, so the hit rate on the highest-value queries — the repetitive FAQ traffic the cache exists to absorb — is close to zero.

The technique that works is semantic caching, and the striking thing is that every component it needs is already in this architecture:

1. Embed the query               <- the RAG pipeline already does this
2. Search cached query embeddings <- a vector index; the design already has one
3. If similarity > threshold, return the cached response
4. Otherwise generate, then cache the (embedding, response) pair

The design has an embedding service and a vector database, and never composes them into a cache. The parts are present and the connection is missing.

The trade-off to state, because it is real: the similarity threshold is a correctness knob.

THRESHOLD TOO LOW:  "return policy" matches "refund policy"
                    -> the WRONG cached answer is served, confidently
THRESHOLD TOO HIGH: near-duplicates miss -> back to exact-match hit rates

And two things must never be semantically cached: personalized answers (anything involving this user's orders or account) and anything with a function call, since a cached "your order shipped Tuesday" is wrong for everyone else. Cache the impersonal tier only — which maps exactly onto the router's "simple/FAQ" class.

When queries are natural language, cache on meaning rather than on the string — and note this is the same lesson as that building block's typeahead, where the API was impersonal precisely so responses were cacheable. Here, the router is what separates the cacheable traffic from the rest.

Prompt caching is a second, cheaper win that needs no similarity threshold

Distinct from response caching and worth knowing, because it exploits the prefill/decode split from Lesson 3.

Every request to this system carries a large, identical prefix:

System instructions          ~200 tokens   <- IDENTICAL on every request
Retrieved chunks           ~2,000 tokens   <- varies
Conversation history       ~1,000 tokens   <- shared across turns in a session
User query                    ~30 tokens   <- varies

Modern inference servers cache the computed attention state (the KV cache) for a shared prefix, so the prefill work for those tokens is done once rather than per request.

Two wins that follow directly:

A shared system prompt is prefilled once, across all users.

Within a session, turn N's prompt is turn N−1's prompt plus new text — so the history portion is already cached and only the delta needs prefilling. That directly attacks the growth problem Lesson 8 raises about conversation length.

Prompt caching reduces TTFT and cost with no correctness risk, unlike semantic response caching, which trades a similarity threshold against wrong answers. Order the two accordingly: prompt caching is free, semantic caching is a judgement call.

The cheapest token is the one never generated

Every technique above reduces the same quantity, and naming it is the way to keep them straight:

Routing:          fewer tokens generated by the EXPENSIVE model
Semantic caching: zero tokens generated at all
Prompt caching:   fewer tokens PREFILLED
Batching:         more tokens per GPU-second
Shorter outputs:  fewer tokens DECODED

Since decode dominates cost and scales with output length, the largest single lever is answer length — and it is the one nobody tunes. A support answer that is three paragraphs where one would do costs three times as much and takes three times as long, forever, on every query.

That is the direct analogue of that building block's conclusion that the client was the biggest capacity lever: the cheapest request is the one that never leaves the browser, and here the cheapest token is the one never generated.

Instruct for brevity in the system prompt and cap max_tokens — it is a one-line change with a permanent, multiplicative effect on both cost and latency.

Next: function calling, and the attack it opens.