GPU Memory: Weights and Precision

In one line: multiply parameters by bytes per parameter, compare to roughly 80GB, and the answer decides whether you are deploying a process or a distributed system.

The calculation

Weight memory  =  parameters  x  bytes per parameter

That is the whole formula. At FP16 it collapses to twice the parameter count in gigabytes.

Model	FP16 weights	Fits on one 80GB card?
7B	14 GB	Yes, with plenty of room
13B	26 GB	Yes
34B	68 GB	Barely — and no room for cache
70B	140 GB	No — needs at least 2
175B	350 GB	No — needs at least 5

The "barely" row is the instructive one. A 34B model technically fits in 80GB and leaves about 12GB, which after runtime overhead is almost no KV cache — meaning a concurrency of roughly one. Fitting the weights is necessary and nowhere near sufficient.

When it does not fit

Two ways to split a model across cards, and they solve different problems.

	Tensor parallelism	Pipeline parallelism
Splits	Each layer across GPUs	Different layers onto different GPUs
Communication	Every layer, every token — very chatty	Only at stage boundaries
Needs	A fast interconnect within one node	Tolerates slower links across nodes
Weakness	Does not scale past one node well	Bubbles — stages idle waiting for work
Use for	Latency-sensitive serving	Very large models, throughput workloads

The practical rule: tensor parallelism inside a node, pipeline parallelism across nodes. Tensor parallelism exchanges activations at every layer for every token, so it needs the high-bandwidth interconnect that exists within a server and does not exist between servers.

There is a cost to naming here: splitting a model does not just add hardware, it adds a failure mode. Every request now depends on several GPUs and the link between them, so the availability of a served request is the product of their availabilities rather than one machine's.

The quantisation ladder

Quantisation reduces bytes per parameter, and it is the first lever to reach for when a model nearly fits.

Precision	70B weights	Rough quality cost
FP16	140 GB	Baseline
FP8	70 GB	Small on recent hardware
INT8	70 GB	Small, well-established
INT4	35 GB	Noticeable, task-dependent

The step from FP16 to INT8 is often close to free in quality and halves the memory — which can be the difference between two cards and one, and therefore between a distributed deployment and a simple one. That is a much larger architectural saving than the memory number alone suggests.

INT4 is where the trade becomes real. It is defensible for some workloads and it needs evaluating rather than assuming, and saying so is more credible than either dismissing or endorsing it.

Serving memory is not training memory

A common confusion worth clearing, because it makes people overestimate serving hardware by roughly an order of magnitude.

Training must hold the weights, the gradients, the optimiser state, and the activations for backpropagation. With a common optimiser that is roughly 16 bytes per parameter rather than 2 — weights, gradients, and two optimiser moments, several of them in FP32.

Training a 7B model:   7e9 x ~16 bytes  ~=  112 GB
Serving a 7B model:    7e9 x 2 bytes    ~=   14 GB

Serving holds only the weights plus the KV cache for in-flight requests. That is why a model you cannot train on one card may serve happily on it, and it is why fine-tuning approaches that avoid holding full optimiser state exist at all.

What is left over is the real question

Once the weights are placed, the number that decides your capacity is what remains.

80 GB per card
- 14 GB weights (7B at FP16)
-  ~5 GB runtime, framework, fragmentation
= ~61 GB available for KV cache

That leftover is the entire budget for concurrency, and how many requests it buys depends on how long they are. That is the next lesson, and it is where most capacity surprises come from.

Key takeaway

Weight memory is parameters times bytes per parameter — at FP16, twice the parameter count in gigabytes — and the first question is whether that fits on one roughly-80GB card, because the answer turns a process into a distributed system. Tensor parallelism belongs inside a node and pipeline parallelism across nodes. Quantising to INT8 often halves memory for little quality, which can eliminate a whole GPU, but it produces a different model that must be re-evaluated on the hard slice. And serving memory is about 2 bytes per parameter where training is nearer 16 — do not size one from the other.

Next: the KV cache, and why concurrency is a memory question.