Cost, Latency and Evaluation

In one line: the median task is cheap and the tail is not, and you cannot tell whether the system worked by looking at what it said.

The tail is the cost

A single-turn system has a cost distribution with a modest spread. An agent's is heavy-tailed by construction: the median task takes a few steps and the worst takes the entire budget.

That has a specific consequence for planning capacity. Averages are useless here. A median of four steps and a p99 of forty means the fleet has to absorb tasks that cost ten times the typical one, and those are exactly the tasks arriving when something is going wrong.

Three levers, in order of effect:

Reduce the step count. By far the largest. Every step is a model call whose prompt has grown, so removing two steps from a six-step task saves more than optimising the model. Better tools that do more per call, better plans that avoid dead ends, and deterministic code replacing steps that were never judgement calls.

Reduce cost per step. The context grows with every observation, so a late step is much more expensive than an early one. Summarise old observations, drop what is no longer relevant, and keep only recent steps verbatim.

Route by step type. Not every step needs the largest model. Deciding which of three tools to call is a much smaller problem than synthesising a final answer, and a small model handles most steps competently.

Latency is a sum, so hide it

Total time is the sum of steps, and there is no way to make it competitive with a single call. So the design accepts it and changes what the user experiences.

Stream progress, not tokens. For a multi-step task the useful stream is the plan and which step is running, not the model's words.

Return partial results. If three of five things are done, show them rather than waiting.

Go asynchronous above a threshold. Past some duration the right interaction is not waiting at all — accept the task, work in the background, notify on completion. That converts a latency problem into a proactivity problem, which is a much better one to have.

That third move is the one that makes long tasks viable, and it depends entirely on the durable state from the previous lesson.

Evaluating a trajectory

The hard part, and it is genuinely different from every other chapter.

Elsewhere the output is the thing: a ranking, a classification, an answer. Here the output is a sequence of actions and a changed world, so success is a property of the trajectory rather than of a final string.

Four things to measure, and they are not substitutes.

Task success. Did the goal get achieved? Requires a checkable definition of achieved, which is why plans should carry success criteria — verified against the world rather than by asking the model.

Efficiency. How many steps, against how many were necessary. A task done in twelve steps that needed four succeeded and was expensive, and that gap is where most improvement lives.

Safety. Did it take an action it should not have? A single serious violation outweighs a great deal of success, so this is a gate rather than an average.

Trajectory quality. Were the steps sensible, or did it stumble into the right answer? This is the one that predicts whether the system will work on tasks you have not tested, and the only one that needs human review.

Building the evaluation set

Live traffic will not give you this, for the same reason as enterprise search: not enough volume, and the interesting cases are rare.

So construct it, and the shape is specific to agents.

Tasks with verifiable outcomes. "Schedule a 30-minute meeting with X next week" is checkable against the calendar. "Help me plan my week" is not, and untestable tasks belong in human review rather than the automated set.

A sandbox that behaves like the world. Agents act, so the eval needs an environment with a calendar, a mailbox and tools that can be inspected and reset. Evaluating against production is not an option once writes are involved.

Deliberate failure injection. A tool that times out, a result that contradicts the plan, an ambiguous match. These are the cases that exercise the recovery logic, which is the part most likely to be broken and least likely to appear in a happy-path suite.

Adversarial content. A calendar invite whose description contains an instruction. An email asking the assistant to forward something. This is where the untrusted-observation rule is tested, and it will not show up by accident.

Observability

What to record, and it is per-step rather than per-request.

Every step: the plan at that moment, the tool and its arguments, the result, the cost, and the reason the loop continued. Without the arguments and results, a failed trajectory is unreadable.

Three things this makes possible. Debugging a specific failure, which is otherwise guesswork. Aggregate analysis — which tools fail most, where trajectories tend to go wrong, which task types blow their budget. And replay, which is how a fix is verified against the case that motivated it.

The signals worth alarming on: budget-exhaustion rate, which is the direct measure of tasks that failed to converge; repetition-detector firing rate, which is stuck loops; and step count by task type, where a rise means something upstream changed — a tool got slower, a plan got worse, or the world moved.

The honest position

Worth stating, because it is true and it is the sort of thing that reads as experience.

Agents are harder to evaluate than anything else in this course. There is no click-through, no chargeback, no relevance judgement — the equivalent artifact is a trajectory through a changing world, and constructing enough of those to be confident is expensive.

So the realistic position is a small, high-quality evaluation set covering the important task types, a safety suite that must pass, per-step observability for diagnosis, and human review of sampled trajectories. That is a weaker guarantee than a ranking system gets, and pretending otherwise — by quoting a task success rate as though it settled the question — is the failure to avoid.

Next: the whole thing, as an interview.