What the Agent Sees

In one line: the page has to become an observation small enough to reason over, and the three ways of doing that fail differently.

Three representations

A screenshot. What a person would see. Layout, emphasis and visual convention are all preserved, which is exactly the information the markup loses. And the model must infer structure from pixels: which region is clickable, what a control does, whether something is disabled.

Raw HTML. Complete and unusable. A modern page is enormous, most of it is framework scaffolding and styling, and the meaningful elements are buried in nesting that carries no semantics. It also consumes context at a rate that makes multi-step tasks unaffordable.

The accessibility tree. The browser's own distillation of the page into roles, names and states — button, link, textbox, with their labels and whether they are enabled. It exists because screen readers need exactly what an agent needs: what is here and what can I do with it, without the visual noise.

The accessibility tree is the strongest single choice, and it is not sufficient on its own — a complex page can still produce a tree of tens of kilobytes, and some meaning genuinely lives only in the rendering.

Why not just one

The published web-agent work has gone both ways and converged on combining them, for a specific reason: the two representations are wrong about different things.

A screenshot shows that a banner is covering the button. The tree does not. The tree shows that a control is disabled and gives its accessible name. The screenshot shows a greyed rectangle the model has to interpret.

So the practical observation is both — a screenshot for layout and visual state, plus the tree for structure and identity — with the two aligned so the model can refer to the same element in either. That alignment is the next lesson.

The size problem

Which is the constraint that shapes everything.

Even the tree is too big on a real page, and the observation is sent on every step. A ten-step task pays it ten times, and each time it competes with the loop's accumulated history for the same context window.

Four reductions, in order of value:

Interactive elements only. Most of a page is text and structure the agent will not act on. The elements that matter are buttons, links, inputs, selects and anything with a handler. Filtering to those cuts the observation dramatically.

Viewport first. What is on screen is what can be clicked. Off-screen content is reachable by scrolling, which is an action, so it does not need to be in this observation.

Collapse the repetitive. Search results and product grids are the same structure fifty times. Summarising as a list with a few representatives is far cheaper and loses almost nothing.

Drop the invisible. Hidden elements, zero-size nodes and things behind an overlay are not actionable, and including them invites the agent to try.

That last one is a correctness fix as much as a size one — an agent that clicks an element hidden behind a modal produces exactly the silent no-op that the verification lesson exists to catch.

The trade you are making

Worth stating explicitly, because every reduction can remove the thing you needed.

Filtering to interactive elements loses the text that explains what they do. Viewport-only loses an element just below the fold. Collapsing repetition loses the specific item the task was about.

There is no reduction that is safe in general, which is why the agent needs a way to ask for more: expand this region, scroll and re-observe, read the full text of this section. Making the observation adjustable is better than trying to find one compression that always works.

Cost, concretely

The arithmetic that makes this a design decision rather than an implementation detail.

A screenshot is a fixed and substantial number of tokens per step, and it recurs every step. A filtered accessibility tree is smaller but grows with page complexity. The loop's history grows on top of both.

Two consequences.

Do not send the screenshot every step. Many steps are decidable from the tree alone — typing into a field whose label you already know does not require looking at the picture. Reserve the visual for steps where layout matters or where the tree was ambiguous.

Do not carry old observations. The page from three steps ago is not needed and is large. Keep the current observation and a short summary of what happened, not a transcript of every page visited.

That second point is the single largest cost lever in a browsing agent, and it is easy to get wrong because carrying history feels safer.

What to record

For debugging, which is disproportionately hard here.

Every step should persist the URL, the reduced observation, the chosen action, and — critically — a screenshot at the moment of acting. When an agent does something inexplicable, the picture of what it was looking at is the only thing that explains it, and it cannot be reconstructed later because the page has moved on.

That is expensive to store and it is the difference between debugging a browsing agent and guessing about one.

Next: turning that observation into an action on a specific element.