Training and Fine-Tuning Economics
In one line: the arithmetic of training exists mostly to show you that you should not be doing it, and knowing the numbers is how you defend that conclusion.
The 6ND rule
Training compute has a simple approximation:
FLOPs ~= 6 x N x D N = parameters D = training tokens
The 6 decomposes cleanly and it is worth being able to explain rather than assert. A forward pass costs about 2N per token — one multiply and one add per parameter. The backward pass costs about twice the forward. Two plus four is six.
That decomposition also gives you inference for free: a forward pass is roughly 2N FLOPs per token, which is the number behind every serving throughput estimate.
What that means for training from scratch
A 7B model, Chinchilla-optimal at 20 tokens per parameter: D = 7e9 x 20 = 1.4e11 tokens C = 6 x 7e9 x 1.4e11 = ~5.9e21 FLOPs
Convert to hardware time with a realistic utilisation figure — accelerators do not hit peak, and 40% is a reasonable planning assumption:
At ~4e14 effective FLOPs/sec per GPU:
5.9e21 / 4e14 = ~1.5e7 GPU-seconds
= ~4,000 GPU-hours
= ~170 GPU-days
So a 7B model is roughly a few thousand GPU-hours — a real but survivable number for a well-funded team. Now scale it: compute grows with N × D, and Chinchilla ties D to N, so cost grows with roughly the square of model size. A 70B model is not ten times a 7B; it is nearer a hundred times.
The Chinchilla ratio
The empirical finding that reset the field's intuitions: for a fixed compute budget, the optimal split is roughly 20 tokens per parameter, and model size and data should scale together rather than one racing ahead.
Its practical consequence is that many earlier large models were undertrained — too big for the data they saw. A smaller model trained on more tokens beats a larger one trained on fewer, at equal compute.
That matters for a system designer for one reason above all: a smaller, well-trained model is cheaper to serve forever. Training cost is paid once; serving cost is paid on every request for the life of the product. Given the choice, the model that is smaller at equal quality is almost always the right one.
Fine-tuning changed shape
Full fine-tuning updates every weight, so it needs training-scale memory — roughly 16 bytes per parameter for weights, gradients and optimiser state. For a 7B model that is over 100GB, which is a multi-GPU job for a task that feels small.
Parameter-efficient methods freeze the base model and train a small number of additional weights. Because the optimiser state only covers those few parameters, memory drops to something close to inference plus a little.
| Full fine-tuning | Parameter-efficient (LoRA-style) | |
|---|---|---|
| Trainable weights | All of them | A small fraction — often well under 1% |
| Memory | ~16 bytes per parameter | Close to inference memory |
| Artifact size | A full model copy per task | Megabytes per task |
| Serving many variants | One deployment per variant | One base model, adapters swapped per request |
| Ceiling | Higher, for large distribution shifts | Slightly lower, and usually sufficient |
The row that matters architecturally is the fourth. Serving a hundred fully fine-tuned variants means a hundred model deployments, which is unaffordable. Serving a hundred adapters means one base model with small per-request adapters, which is entirely reasonable — and that is what makes per-customer or per-task specialisation a viable product feature rather than a research demo.
The decision ladder
Almost every "should we train a model?" question resolves further down this ladder than the asker expects. Work top to bottom and stop at the first rung that works.
The distinction that decides between rungs two and three is the one to state clearly:
Retrieval adds knowledge. Fine-tuning changes behaviour.
If the model does not know something — your documentation, this customer's history, yesterday's prices — that is retrieval, and fine-tuning is the wrong tool because facts baked into weights go stale and cannot be updated without retraining. If the model knows the material but responds in the wrong format, tone, or style, that is fine-tuning, and retrieval will not fix it.
Getting this backwards is a common and expensive mistake: teams fine-tune a model on their documentation, find it hallucinates confidently about it, and have no way to correct a fact short of another training run.
Key takeaway
Training compute is roughly 6ND, which decomposes into 2N forward and 4N backward — and gives inference at 2N per token for free. Because Chinchilla ties data to parameters, training cost scales with roughly the square of model size, and only the successful run is in that estimate. Training is paid once and serving forever, so at equal quality take the smaller model. Parameter-efficient fine-tuning matters most for its artifact size, which makes one base model with swappable adapters possible. And work the ladder top-down: retrieval adds knowledge, fine-tuning changes behaviour, and confusing the two is the expensive mistake.
Next: turning all of this into a cost model, and the build-versus-buy line.