Cost, Latency and Evaluation

In one line: the expensive part is looking, the slow part is waiting, and you cannot test any of it against a web that changes underneath you.

Where the money goes

Unlike the previous chapter, where reasoning dominated, here it is perception.

Every step sends an observation, and an observation is a screenshot plus a filtered tree. Both are large, both recur per step, and both sit on top of a history that is also growing.

Three levers, in order of effect.

Do not send the screenshot every step. Many steps are decidable from the tree alone — typing into a field whose label is already known needs no picture. Reserve the visual for steps where layout matters or the tree was ambiguous. This is usually the single largest saving available.

Reduce harder. Interactive elements only, viewport first, collapse repetition, drop the invisible. Every element removed is paid for on every subsequent step it would have appeared in.

Do not carry old observations. Keep the current page and a short summary of what happened, not a transcript of everything visited. This is the same point as the previous chapter and it bites harder here because each observation is bigger.

Where the time goes

Not the model. Page loads, waits and network are seconds each, and a browsing step is dominated by them.

Which produces a different optimisation profile from every other system in this course: making the model faster barely helps. A ten-step task with three-second page loads is thirty seconds of waiting before any inference.

Three things that do help.

Fewer steps. The same conclusion as everywhere, and it matters more here because each step carries a page load.

Parallel exploration. Comparing four sites is four independent trajectories, and running them concurrently turns a sum into a maximum. This is the largest latency win available and it only works for read-only work.

Reusing the browser. A warm context avoids cold-start, at the cost of the isolation the scoping lesson argued for — which makes it a trade rather than a free win, and the safe version is pooling within a single user's tasks rather than across users.

And the honest conclusion: browsing is slow, so above a threshold the right interaction is asynchronous. Accept the task, work in the background, notify on completion — the same move as the previous chapter, and more necessary here.

Evaluating against a moving target

The genuine difficulty, and it is worse than in any other chapter.

The live web changes. A test that passed yesterday fails today because the site redesigned, the prices moved, or the result ordering changed. So a failing test means either the agent regressed or the world did, and the two are indistinguishable without looking.

Four responses, and a real system uses all of them.

Recorded sessions. Capture real pages and replay them deterministically. Fast, free, repeatable, and it goes stale — it tests against the web as it was.

A sandbox site. A controlled environment that behaves like a real site and never changes without your say-so. This is what published web-agent benchmarks provide, and it is the only way to get a stable number.

Live tests, sampled and tolerant. A small suite against real sites, expected to be noisy, watched for trend rather than treated as pass or fail.

Human review of trajectories. Expensive and irreplaceable, particularly for the wrong-page failure that no automated check catches.

What to measure

Success alone is not enough, for the same reason as the previous chapter — a trajectory can succeed expensively or by luck.

Task success, verified against evidence on the page rather than the agent's assertion.

Steps taken versus necessary. The gap is where improvement lives, and in browsing the gap is usually mis-clicks, dismissals and re-observations.

Verification failure rate — how often a step's expectation was not met. This is the best single health metric here, because it rises immediately when a site changes, when the reduction dropped something needed, or when waits are too short.

Recovery rate. When it did get stuck, how often did it get out. A system with a high stuck rate and a high recovery rate is in much better shape than the numbers alone suggest.

Human handoff rate, split by reason. A rise in login handoffs is a different problem from a rise in ambiguity handoffs.

That third metric is the one to build the dashboard around. It is cheap, it is per-step, and it moves before task success does — which makes it the early warning that a site redesign has broken something.

The eval set

What it needs to contain, and two of these will not appear by accident.

Tasks with verifiable outcomes across a range of sites and difficulties.

Sites that are hard in specific ways — heavy single-page apps, infinite scroll, aggressive modals, multi-step forms. Each exercises a different part of the perception and waiting machinery.

Injected adversarial pages. A page containing text addressed to the agent, including hidden content present only in the tree. This tests the trust boundary, and it is the case a happy-path suite never produces.

Deliberate breakage. A slow-loading page, an element that moves after render, a redirect mid-task. These exercise the recovery paths, which are the least-tested and most-likely-broken part of the system.

Observability

Per step, and this chapter needs more of it than any other.

URL, the reduced observation, the chosen label and what element it resolved to, the action, the stated expectation, the verification result, and a screenshot at the moment of acting.

That screenshot is not optional. When an agent does something inexplicable, the picture of what it was looking at is the only artifact that explains it, and it cannot be reconstructed later because the page has moved on. Everything else in the trace tells you what happened; the screenshot tells you why.

Next: the whole thing, as an interview.