Estimation Drills

In one line: do each one in under ninety seconds before reading the answer, and check the conclusion rather than the digits.

Tokens

1. A 40-page document is passed to a model. Roughly how many tokens?

~500 tokens per page × 40 = ~20,000 tokens. Under a typical long-context limit, and enough that you would not send it on every turn of a conversation — this is a candidate for retrieval rather than stuffing.

2. A chat runs 15 turns, each exchange adding ~400 tokens. Total tokens sent across the conversation?

t × N × (N+1) / 2 = 400 × 15 × 16 / 2 = 48,000 tokens. The naive answer, 400 × 15 = 6,000, is eight times too low. The decision: cap or summarise history.

3. 500,000 requests a day, 2,000 prompt and 200 completion tokens. Monthly token volume?

Per day: 500,000 × 2,200 = 1.1e9. Monthly: ~33 billion tokens. The split matters — 30 billion prompt against 3 billion completion means prompt caching and context trimming are where the money is.

GPU memory

4. A 13B model at FP16. Does it fit on one 80GB card, and what is left?

13e9 × 2 bytes = 26GB. Yes. After ~5GB runtime overhead that leaves ~49GB for KV cache — comfortable, and the real question becomes concurrency.

5. A 70B model at INT8. How many cards?

70e9 × 1 byte = 70GB. It fits on one 80GB card for weights — but that leaves almost nothing for cache, so practically two, giving room to actually serve requests. Fitting the weights is necessary, not sufficient.

6. Per-token KV cache for a model with 60 layers, 8 KV heads, head dimension 128, at FP16?

2 × 60 × 8 × 128 × 2 = 245,760 bytes ≈ 0.23 MB per token. At 8,000 context that is ~1.9GB per in-flight request.

7. Same model, 50GB free for cache, 8,000-token contexts. Max concurrency?

50 / 1.9 = ~26 concurrent requests. If you need 200, that is 8 GPUs on memory alone — and the decision is to reduce context or accept the fleet.

Concurrency and throughput

8. 40 requests per second, average 6 seconds each. Concurrency?

Little's law: 40 × 6 = 240 concurrent requests. This is the number to divide into per-GPU capacity, and candidates routinely skip it and size on request rate instead.

9. A GPU sustains 2,000 tokens/sec batched decode. Average completion is 400 tokens. Completions per second?

2,000 / 400 = 5 per second. For 40 per second you need 8 GPUs on throughput — then compare against the memory bound and take the larger.

Retrieval

10. 200,000 documents, ~8 pages each, 2 chunks per page, 1,536 dimensions, float32. Index size in memory?

Chunks: 200,000 × 8 × 2 = 3.2M. Raw: 3.2e6 × 1,536 × 4 = ~20GB. With HNSW overhead at 1.75×: ~35GB. Quantised to int8: ~9GB. The decision: it fits in memory comfortably once quantised, so no sharding needed yet.

11. The same corpus at 768 dimensions instead. What changes?

Everything halves — ~10GB raw, ~17GB indexed, ~4.5GB quantised. Worth stating that embedding dimension is a capacity decision, and that the recall difference between 768 and 1,536 is often small enough to be worth measuring before paying double forever.

Cost

12. 33 billion tokens a month, 90% prompt at $3/M and 10% completion at $15/M. Monthly cost?

Prompt: 29.7e9 / 1e6 × $3 ≈ $89,000. Completion: 3.3e9 / 1e6 × $15 ≈ $50,000. Total ≈ $139,000/month.

Two decisions fall out. Completion is 10% of tokens and 36% of cost, so shorter answers are worth more than they look. And routing 80% of traffic to a model ten times cheaper saves roughly $100,000 a month — which is why routing is the first lever, not the last.