Numbers Worth Memorising
In one line: a handful of anchors turns a two-minute derivation into a ten-second one, and knowing them is the difference between estimating live and stalling.
Precision, in bytes per parameter
Everything about model memory starts here.
| Precision | Bytes | Typical use |
|---|---|---|
| FP32 | 4 | Rarely used for inference now |
| FP16 / BF16 | 2 | The serving default |
| FP8 | 1 | Increasingly common on recent hardware |
| INT8 | 1 | Quantised serving |
| INT4 | 0.5 | Aggressive quantisation, noticeable quality cost |
The one to internalise: FP16 means bytes equals twice the parameter count. A 7B model is 14GB of weights. A 70B model is 140GB. You will use this constantly.
Accelerator capacity
You do not need a spec sheet, you need one anchor and the ability to scale from it.
H100 80 GB HBM A100 40 or 80 GB Anchor: ~80 GB per card
From that single number, useful conclusions fall out immediately. A 70B model at FP16 needs 140GB of weights, which does not fit on one 80GB card — so it needs at least two, with tensor parallelism, before a single request has been served. A 7B model at 14GB fits comfortably with room left for cache.
That "does it fit on one card" question is the first thing to answer about any self-hosted model, because the answer changes the deployment from a process into a distributed system.
Tokens
| Anchor | Value |
|---|---|
| Tokens per English word | ~1.3 |
| Words per token | ~0.75 |
| Tokens per page of prose | ~500 |
| Tokens in a typical chat turn | 50–200 |
| Tokens in a RAG prompt with retrieved context | 2,000–8,000 |
The ratio to remember is roughly 4 characters per token, or three-quarters of a word. Everything else can be derived from it live.
Embedding dimensions
| Model class | Dimensions |
|---|---|
| Small / efficient | 384 |
| Common general-purpose | 768 |
| Large | 1,024–1,536 |
| Very large | 3,072 |
Use 768 as the default anchor and 1,536 when someone says "large". Combined with the bytes-per-value figure, this gives vector storage instantly: a million 768-dimension float32 vectors is 1e6 × 768 × 4 ≈ 3GB before any index overhead.
The latency ladder, AI edition
The classic ladder still holds underneath. These are the operations the classic one does not have.
The shape to notice: everything before generation is measured in milliseconds, and generation is measured in seconds. That is why streaming exists — not as a nicety but because time-to-first-token is the only latency number a user actually experiences, and it can be two orders of magnitude better than time-to-last-token.
Throughput anchors
Rough, hardware- and model-dependent, and enough to reason with:
| Quantity | Anchor |
|---|---|
| Single-stream decode, 7B class | ~50–100 tokens/sec |
| Single-stream decode, 70B class | ~10–30 tokens/sec |
| Batched throughput gain | 10× or more over single stream |
| Human reading speed | ~5 tokens/sec |
That last row is the useful one and it is easy to forget. A model generating 30 tokens per second is producing text about six times faster than a person reads it — so for a chat interface, generation speed past a certain point buys nothing, and the budget is better spent on time-to-first-token or on cost.
Price anchors
Prices move, so memorise the shape rather than the figures.
Small hosted model ~$0.1-0.5 per million tokens Frontier hosted model ~$3-15 per million tokens Spread between tiers roughly 10-30x Output vs input output typically 3-5x the input price
The spread is the number that matters, because it is what makes routing worth designing. If cheap and expensive models differ by 20× and 80% of your traffic is simple, routing that 80% to the cheap tier cuts cost by roughly 75% — which is a bigger win than almost any other optimisation available.
Storage anchors
| Item | Size |
|---|---|
| One float32 value | 4 bytes |
| One 768-dim float32 vector | ~3 KB |
| One million such vectors, raw | ~3 GB |
| HNSW index overhead | 1.5–2× the raw vectors |
| A page of text | ~2 KB |
Key takeaway
Memorise the shape rather than the spec sheet: FP16 means two bytes per parameter, an accelerator holds roughly 80GB, a token is about three-quarters of a word, a default embedding is 768 dimensions, and hosted model tiers differ by 10–30× with output priced several times input. From those five, every other number in this chapter is a short derivation — and the price spread is what makes routing the highest-leverage cost optimisation available.
Next: token arithmetic, and why the prompt is usually the expensive half.