Cost Modelling and Build vs Buy
In one line: build the cost per request first, because every architectural argument in an AI system eventually reduces to it — and the self-hosting break-even is a utilisation number, not a volume number.
Cost per request, from the bottom
Take the support assistant and price one interaction properly.
Per turn:
prompt 3,000 tokens @ $3.00 / 1M = $0.0090
completion 250 tokens @ $15.00 / 1M = $0.0038
--------
$0.0128
Per 8-turn conversation, history included:
~8 x average 3,250 tokens, growing ≈ $0.13
Per month, 1M conversations ≈ $130,000
Now do the same on a small model at roughly a tenth the price and it is about 13,000 dollars. The gap is the budget available for every optimisation in the rest of the system, and quantifying it is what makes routing a business decision rather than an engineering preference.
The three cost structures
They behave differently as you grow, and choosing between them is really a choice about which curve you want.
| Hosted API | Self-hosted | Fine-tuned small model | |
|---|---|---|---|
| Cost shape | Purely variable, per token | Mostly fixed, per GPU-hour | Fixed training, then low variable |
| At low volume | Very cheap | Terrible — you pay for idle | Bad — training does not amortise |
| At high volume | Expensive and linear | Much cheaper per token | Cheapest, if quality holds |
| Time to ship | Hours | Weeks | Weeks plus data work |
| Operational load | Near zero | A GPU fleet to run | A fleet plus a retraining pipeline |
The break-even is about utilisation
The common framing is "at what volume does self-hosting win?", and that is subtly the wrong question. A GPU costs the same whether it is busy or idle, so what actually decides the comparison is how full you can keep it.
One GPU at ~$2/hour = ~$1,460/month Sustained throughput = ~2,000 tokens/sec batched If kept 100% busy = ~5.2e9 tokens/month Effective cost = ~$0.28 per million tokens At 30% average utilisation = ~$0.94 per million tokens At 10% average utilisation = ~$2.80 per million tokens
Against a hosted small model at a few tenths of a dollar per million, self-hosting only wins at high sustained utilisation. Against a frontier model at several dollars per million, it wins much earlier.
That closing point is the genuinely useful one. An interactive product alone has a peaky demand curve and poor average utilisation. Give the same fleet an overnight batch job — enrichment, re-embedding, evaluation runs — and average utilisation rises without buying anything, which can move the break-even more than any model optimisation.
The costs that appear in neither column
An estimate that stops at inference is incomplete, and naming these is a maturity signal.
| Cost | Why it is missed |
|---|---|
| Embedding the corpus | A one-off that is large: millions of chunks through an embedding model |
| Re-embedding on model change | Changing embedding models means rebuilding everything |
| Evaluation runs | Every candidate change runs the eval suite — this is real inference spend |
| Retries and escalations | A cascade that escalates 10% of traffic pays twice on that slice |
| Failed and abandoned requests | Tokens generated for a user who navigated away are still billed |
| Storage and egress | Vector indexes, logged prompts and responses, and moving them |
The second row deserves emphasis because it constrains a decision people make lightly. Switching embedding models is not a config change — it invalidates every vector you hold and requires a full re-embed and re-index of the corpus. That makes the initial embedding model choice semi-permanent, and it is worth flagging when you choose one.
Where the money actually goes
A rough shape for a retrieval-augmented product at scale, useful for knowing where to look first:
Generation (LLM inference) ~70-85% Embedding (query + ingest) ~5-10% Vector search infrastructure ~5-10% Everything else ~5%
Generation dominates, which is why the levers from the token lesson — routing, caching, context trimming — are where the effort goes. Optimising vector search to save 20% of 8% is not where the money is, and being able to say that shows you have prioritised rather than enumerated.
Presenting it live
A sequence that works under time pressure:
State the traffic. "A million conversations a month, eight turns, so roughly eight million model calls."
Price one unit. "About three thousand prompt tokens and two-fifty completion per turn — around a cent and a bit."
Scale it and round. "So order of a hundred thousand a month on a frontier model."
Name the lever and its size. "That's why I'd route — the tier spread is ten to thirty times, and if 80% of traffic is simple that's most of the bill."
Give the sensitivity. "It's dominated by conversation length, and that grows quadratically — so capping history is the highest-leverage thing we can do."
That is five sentences and it demonstrates the whole chapter.
Key takeaway
Build cost per request from the bottom and always give a per-unit figure someone can compare to what the unit is worth. The build-versus-buy line is set by average utilisation rather than volume, because a GPU costs the same idle — which is why colocating batch and interactive workloads can decide it outright. Include the costs that sit outside inference, especially corpus embedding and the fact that changing embedding models invalidates every vector you hold. Generation is 70–85% of the bill, so optimise there, and monitor cost per request continuously because it drifts upward on its own.
Next: running the whole estimate live.