Cheat Sheet

Key takeaway

Three shifts from classic estimation: the unit is the token, the binding resource is GPU memory, and cost is a first-order constraint. Everything else is classic capacity planning underneath.

The formulas

Weight memory      =  parameters x bytes per parameter
                      (FP16 -> 2 x param count, in GB)

KV per token       =  2 x layers x kv_heads x head_dim x bytes

Cache per request  =  KV per token x context length

Max concurrency    =  free memory / cache per request

Conversation cost  =  t x N x (N + 1) / 2   ~=  t x N^2 / 2

Vector storage     =  vectors x dimensions x bytes
Index in memory    =  raw x 1.5 to 2        (HNSW graph overhead)

Training FLOPs     =  6 x N x D             (2N forward, 4N backward)
Inference FLOPs    =  2 x N per token

Concurrency        =  rate x duration       (Little's law)

Anchors

Quantity	Value
FP16 / BF16	2 bytes per parameter
INT8	1 byte per parameter
Accelerator memory	~80 GB
Tokens per word	~1.3 (≈ 4 chars per token)
Tokens per page	~500
Default embedding dims	768 (large: 1,536)
One float32 value	4 bytes
Decode, 7B class	~50–100 tok/sec single, ~2,000 batched
Human reading speed	~5 tok/sec
Model tier price spread	10–30×
Output vs input price	3–5×
Chinchilla ratio	~20 tokens per parameter
GPU hourly cost	~$2

Does it fit?

Model	FP16	One 80GB card?
7B	14 GB	Yes, ~61 GB left for cache
13B	26 GB	Yes
34B	68 GB	Barely — no cache room
70B	140 GB	No — 2+ with tensor parallelism

Tensor parallel inside a node, pipeline parallel across nodes.

Context length is the biggest lever on fleet size

Long context is a fleet-sizing decision, not a feature toggle. Route the few requests that need it to a separate pool.

Prefill vs decode

	Prefill	Decode
Parallelism	Whole prompt at once	One token at a time
Bound by	Compute	Memory bandwidth
Sets	Time to first token	Tokens per second
Batching helps	Barely — already saturated	Enormously — weight read amortised

Continuous batching schedules per iteration, so a finished request frees its slot immediately. Paged cache recovers the 60–80% that contiguous reservation wastes. Colocated phases mean a long prompt stalls everyone — fix with chunked prefill, or disaggregate at scale and pay the cache transfer.

Vector index

Method	Size vs float32	Recall cost
float16	1/2	Negligible
int8 (scalar)	~1/4	Small — the default
int4	~1/8	Moderate
Product quantisation	~1/10 or less	Real — pair with rerank

Count chunks, not documents (10–20× multiplier). Past ~100M vectors, in-memory stops being economical — shard first if the corpus partitions, disk-based only if it does not. Changing embedding model invalidates every vector: treat the choice as semi-permanent.

The adaptation ladder

Serving cost is paid forever; training is paid once. At equal quality, take the smaller model.

Cost levers, ranked

Route by difficulty — attacks price per token; tier spread is 10–30×
Cap or summarise history — attacks the quadratic; the only lever that compounds
Cache — exact, semantic (watch the threshold), and prompt-prefix
Trim retrieved context — direct quality trade
Batch — buys throughput, spends latency

Where the money goes: generation 70–85%, embedding 5–10%, vector search 5–10%. Optimise generation.

Build vs buy is decided by average utilisation, not volume — a GPU costs the same idle. Colocating a batch workload with an interactive one can move the break-even on its own.

Quick decision cues

Asked to size a generative system → tokens, split prompt and completion
Multi-turn → quadratic, t × N² / 2
Self-hosted model → does it fit on one card, and what is left for cache
"Support 128K context" → that is a fleet-sizing question; ask what fraction needs it
Sizing a fleet → compute memory bound AND throughput bound, take the larger
Concurrency → Little's law: rate × duration
"One million documents" → ask the chunking strategy first
Index too big → quantise, then shard, then disk
"Should we fine-tune?" → is the gap knowledge or behaviour?
Cost too high → route first, cap history second