Autoscaling a GPU Fleet

In one line: everything that makes autoscaling work for stateless web servers — a good load signal and fast instance start — is absent here, so the design shifts from reacting to provisioning.

Why the usual signals mislead

**CPU utilisation measures the wrong processor. A GPU worker can be saturated while its CPU idles, so CPU-based scaling reacts either never or to something unrelated.

Request rate ignores that requests differ in cost by orders of magnitude. Fifty short completions and one 100K-context summarisation are not comparable load, and a fleet scaled on request count will be wrong in both directions.

GPU utilisation** is closer and still deceptive. During decode a GPU can report high utilisation while being memory-bandwidth-bound with a half-empty batch — busy, and not doing useful work at capacity.

What to scale on

The useful signals are the ones that describe whether requests are being served on time and whether the hardware has room.

Signal	Tells you	Good for
Queue depth and wait time	Demand is exceeding capacity right now	The primary scale-up trigger
Time to first token, p95	The user-visible contract is slipping	The SLO-anchored trigger
Batch occupancy	Whether the engine has room in-flight	Detecting real headroom
KV cache utilisation	The memory constraint on concurrency	The binding limit for long-context traffic
Tokens per second in flight	Actual work rate, cost-weighted	Capacity planning and comparison

Queue wait time is the best single trigger, because it is the thing that directly harms users and it rises before anything else visibly breaks. Pairing it with p95 time to first token gives you a demand signal and an SLO signal, which is a reasonable two-metric answer.

The KV cache row matters for the reason the estimation chapter established: with long contexts, memory binds before throughput does. A fleet serving mixed context lengths needs both signals, because which one saturates first depends on the traffic mix that hour.

The scale-up problem

Even with a perfect signal, the reaction is slow.

Detect sustained pressure          30-60 s
Acquire an accelerator             seconds to MINUTES, if available at all
Pull the container image           1-5 min   (image sizes are large)
Load model weights into GPU        1-5 min   (tens to hundreds of GB)
Warm up, health check              30-60 s
                                   ---------
Total                              ~5-15 minutes

A spike lasting five minutes is entirely over before a new worker serves its first token. So autoscaling does not solve spikes here — it solves sustained shifts in demand, like a daily traffic curve.

Which means spikes must be handled by the mechanisms from the queueing lesson: admission control, degradation and shedding. That connection is the point of this lesson, and it is worth stating explicitly rather than leaving implied.

The third branch is the underrated one. Most products have a demand curve that is roughly the same every weekday, and pre-warming on a schedule beats reacting every single time — you already know the peak is coming at 9am.

Provision for headroom, not for the average

Because reaction is slow, the fleet must carry enough spare capacity to absorb a spike while new workers start.

That headroom is a real cost and choosing it is a business decision: too little and every spike degrades users, too much and you pay for idle accelerators. The honest framing is that you are buying insurance and the premium is visible on the invoice.

Two things reduce the premium without reducing the protection:

**Fill the headroom with preemptible work. Batch jobs, re-embedding, evaluation runs — work that can be paused instantly when interactive demand arrives. The capacity is reserved for spikes and busy the rest of the time, which is the same colocation argument as the estimation chapter and it applies here specifically as a headroom strategy.

Keep a burst path to a hosted API.** When your fleet is saturated, overflow to a provider endpoint rather than shedding. You pay per-token rates for the overflow only, and it converts a capacity failure into a cost spike — which is usually the better trade.

Key takeaway

CPU utilisation measures the wrong processor, request rate ignores that requests differ in cost by orders of magnitude, and GPU utilisation reads high on a half-empty batch — scale on queue wait time and p95 time to first token instead, adding KV cache utilisation when contexts are long. But a scale-up takes five to fifteen minutes, so autoscaling handles sustained shifts and never spikes: absorb those with degradation and shedding, and pre-warm on a schedule where the curve is predictable. Carry headroom, fill it with preemptible work, and drain workers properly on the way down.

Next: sharing a fixed fleet between tenants who all think it is theirs.