Routing by Difficulty

In one line: model tiers differ in price by ten to thirty times and most traffic is easy, so sending easy requests to a cheap model is the biggest cost decision in the system.

The arithmetic that justifies it

1,000,000 requests/month at 3,000 tokens each   =  3e9 tokens

All frontier, ~$5/M                              =  $15,000/month
80% routed to a small model at ~$0.30/M          =  $3,720/month
                                                    ---------
Saving                                              ~75%

Nothing else in the platform moves the bill like that. Caching helps in proportion to repeat rate; context trimming is bounded by what you can safely drop; routing multiplies the price of most of your traffic by a factor of ten.

The precondition is that the traffic distribution is genuinely skewed — and in most products it is. Simple lookups, greetings, format conversions and short factual answers dominate; genuinely hard reasoning is the tail.

Two ways to decide, and one is usually better

Classify first. A cheap classifier predicts difficulty and dispatches accordingly. One extra call, a clean decision point — and it is a model that can be wrong, and it must judge difficulty before seeing any attempt at an answer, which is genuinely hard.

Try cheap, escalate on failure. Send everything to the small model, detect a bad result, and retry on the large one.

Escalation is usually the stronger design, because judging an answer is easier than predicting difficulty. You have the actual output in hand, and many failure signals are free.

The escalation check

What makes this work is that the check is often deterministic and costs nothing:

The output failed to parse when a schema was required.

The model expressed low confidence, refused, or hedged.

A required citation is missing or does not appear in the retrieved context.

**A validator rejected it — the SQL did not run, the code did not compile, the tool call had invalid arguments.

A rule fired** — the answer was implausibly short, or contained a placeholder.

These are the cheap deterministic checks from the evaluation chapter, doing a second job. Where a real judgement is needed, a small judge model is an option, and it adds a call to every request rather than only failures.

The rate that decides whether it pays

Escalation costs double on the escalated fraction, so the economics depend entirely on how often it fires.

p = escalation rate
Cost  =  small_cost  +  p x large_cost

At p = 0.10:  0.30 + 0.10 x 5.00  =  $0.80 per million-token-equivalent
At p = 0.30:  0.30 + 0.30 x 5.00  =  $1.80
At p = 0.60:  0.30 + 0.60 x 5.00  =  $3.30
At p = 1.00:  0.30 + 5.00         =  $5.30   <- worse than not routing at all

The crossover is real and worth naming: above roughly 90% escalation you are paying more than sending everything to the large model directly. So escalation rate is not a curiosity, it is a monitored metric with an alert, and a drifting one means the routing has stopped working.

The latency cost nobody mentions

Escalation makes the tail worse, and this is the honest objection to it.

An escalated request pays the small model's full generation, the check, and then the large model's full generation. It is not slower by a little — it is roughly the sum of two complete requests.

Two mitigations worth knowing. Stream from the small model and only discard on failure — the user sees output immediately, and the escalation restarts. That trades a visible restart for a better perceived latency, and whether it is acceptable is a product call. Or **run both in parallel for a subset and take the better, which costs money to buy back the tail.

What else routing can select

Difficulty is the main axis and not the only one, and mentioning the others shows the mechanism is understood rather than memorised:

Context length. Long-context requests go to a pool sized for them, so the whole fleet is not provisioned for the worst case.

Modality. Image or audio input needs a different model entirely.

Latency class. An interactive request takes the fast small model; a background job takes whatever is cheapest.

Tenant tier.** A paying customer's traffic gets the better model, which is a product decision expressed in routing.

Next: making the output structurally reliable.