Token Arithmetic
In one line: count tokens, split them into prompt and completion, and remember that a conversation re-sends its own history on every turn.
A single request
Start with the simplest possible case and build up. A support question with retrieved context:
System prompt and instructions 400 tokens
Retrieved context, 5 chunks x 400 2,000 tokens
Conversation history 600 tokens
The user's question 50 tokens
------------
PROMPT total 3,050 tokens
COMPLETION 250 tokens
Two observations that candidates routinely miss.
The prompt dominates. It is twelve times the completion here, and that ratio is typical for any retrieval-augmented system. If you are trying to reduce cost, the retrieved context is where the tokens are — not the answer.
The system prompt is paid on every single request. Four hundred tokens sounds trivial until you multiply by a million requests a day, at which point it is 400 million tokens of instructions you are re-sending to a model that has already seen them a million times. That observation is what motivates prompt caching.
Multi-turn conversations grow quadratically
This is the single most important token fact and the one most often got wrong.
A stateless model has no memory, so each turn re-sends the entire conversation so far. Turn 10 does not cost the same as turn 1 — it carries nine turns of history.
Work it through. With roughly 300 tokens per exchange and a 10-turn conversation:
Turn 1 sends 300
Turn 2 sends 600
Turn 3 sends 900
...
Turn 10 sends 3,000
-----
Total 16,500 tokens
Naive estimate: 10 turns × 300 tokens = 3,000. Actual: 16,500. A factor of five and a half, and it gets worse as conversations get longer — at 20 turns the multiplier is roughly eleven.
The general form is worth knowing because it is easy to derive live:
Total tokens over N turns = t * N * (N + 1) / 2
~= t * N^2 / 2
where t = tokens added per turn
Getting this wrong understates cost by five to ten times, which is the difference between a viable product and one that is quietly losing money on every user.
From a request to a monthly bill
Now scale up. A support assistant:
1,000,000 conversations per month x 8 turns each x ~3,000 tokens per turn on average (history included) = 24,000,000,000 tokens per month
At a frontier-model price of roughly 5 dollars per million tokens that is 120,000 dollars a month. At a small-model price of roughly 0.3 dollars per million it is 7,200 dollars.
That spread — a factor of about seventeen — is the entire argument for the routing, caching and context-trimming machinery that shows up in every LLM architecture in this course. It is not premature optimisation; it is the difference between two products.
The three levers
Each attacks a different term in the arithmetic, and a good answer names which one it is pulling.
| Lever | Which term it attacks | Typical saving |
|---|---|---|
| Route by difficulty | The price per token | Large — the tier spread is 10-30x |
| Cache | The number of requests reaching a model | Proportional to the repeat rate |
| Trim the context | Tokens per request | Bounded by what you can safely drop |
Routing
Send easy requests to a small model and hard ones to a large one. Because the tier spread is 10–30×, and because most traffic in most products is easy, this is usually the biggest single win available.
The design question is what does the routing, and the honest answer is that a classifier deciding difficulty is itself a model that can be wrong. The usual resolution is to route optimistically and escalate: try the cheap model, detect low confidence or a failed check, and retry on the expensive one. That costs you double on the escalated fraction, so it only pays while that fraction stays small — which is a number you must monitor.
Caching
Two distinct kinds, and conflating them is a common error.
Exact caching returns a stored response for a byte-identical request. Cheap, safe, and only useful where requests genuinely repeat.
Semantic caching returns a stored response for a request that is similar enough by embedding distance. Far higher hit rate, and it can return a confidently wrong answer when the similarity threshold is loose — "flights to Paris on Tuesday" and "flights to Paris on Thursday" are extremely close in embedding space and have different answers.
Prompt caching is the third and it attacks the system-prompt problem directly: the provider caches the processed prefix so a shared instruction block is not recomputed per request. It requires the stable part of the prompt to come first, which is a prompt-structure decision with a direct cost consequence.
Trimming context
Reduce tokens per request. Retrieve five chunks instead of twenty. Summarise conversation history past turn ten rather than re-sending it verbatim. Truncate documents to the relevant section.
The trade is directly against quality, and the honest framing is that you are choosing a point on a curve rather than getting something free.
That closing point is worth making explicitly. Because conversation cost grows with the square of turn count, capping or summarising history is the one trimming decision whose benefit compounds — it does not save a fixed amount, it changes the growth rate.
Key takeaway
Size generative workloads in tokens, split prompt from completion, and expect the prompt to dominate in any retrieval system. A stateless model re-sends the whole conversation every turn, so total cost grows with the square of turn count — roughly t × N² / 2 — and estimating linearly understates the bill five to ten times. Three levers attack three different terms: routing changes the price, caching changes the request count, and trimming changes tokens per request, with history-trimming the only one whose benefit compounds.
Next: what those tokens require in GPU memory.