The Web Was Not Built for This

In one line: a web page is a rendering intended for human eyes, and the agent has to recover the interface from the picture.

A different hard problem

The personal assistant's difficulty was the loop — planning, terminating, remembering. Those apply here too, and they are not what makes browsing hard.

Browsing is hard because of perception. The agent is operating a medium designed for a person with eyes and a pointer, through an interface that offers either a picture or a tree of markup, neither of which says plainly "here is the button that submits this form".

Three properties of the web make this genuinely difficult.

It is designed for humans. Visual hierarchy, colour, position and convention carry meaning that is nowhere in the markup. A big green button in the bottom right is obviously the primary action to a person, and is a div with three classes to a parser.

It is enormous and mostly irrelevant. A page has thousands of nodes and a handful matter. Sending everything wastes the context; sending the wrong subset loses the element you needed.

It changes under you. Content loads after the page does. Elements move. A single-page app rewrites the DOM without navigating. The thing you decided to click may not exist by the time you click it.

The two questions every step must answer

Which reduces the whole chapter to two problems the loop from the previous chapter does not have.

What am I looking at? Turning a rendered page into something small enough to reason over and specific enough to act on. That is grounding, and it gets two lessons.

Did that work? After clicking, the agent has an assumption about what happened, and no direct confirmation. A click that hit the wrong element, or hit nothing, or triggered something unexpected, all look identical from the agent's side until it looks again.

The second question is the one designs skip. An agent that acts and immediately plans its next step on the assumption the action succeeded is building on something it never checked — and errors compound, because the next observation is interpreted through a belief that is already wrong.

Why not just use the API

The first question to ask, and the answer is usually that you should.

If the site has an API, use it. It is structured, stable, documented, rate-limited in a way you can plan around, and it does not require the agent to interpret a rendering. An agent driving a browser to do what a REST call would do is strictly worse on every axis.

Browsing earns its place in three situations:

There is no API. Common, and the honest reason most of these systems exist.

The task spans sites. Comparing across several vendors, none of which share a schema. The browser is the universal client.

The workflow is the interface. Some things only exist as a sequence of pages — a booking flow, a government form, a legacy internal tool.

Saying this out loud is worth doing. A design that reaches for browser automation without asking whether an API exists has skipped the cheapest available answer.

What browsing buys, and costs

That row on stability is the operational reality. A site redesign breaks an agent silently — no error, just a page whose elements no longer match what it learned to expect — and there is no deprecation notice because nobody knew you were there.

The shape of the answer

Which sets up the rest of the chapter. The loop is the same as the previous one — plan, act, observe, check — with three additions specific to this environment.

Perception, before every step. Turn the page into a compact, actionable observation.

Verification, after every step. State what should change, then confirm it did.

A trust boundary around the page. The content is written by whoever owns the site, which in the general case is an adversary, and it arrives as the observation the agent reasons over. That is indirect injection in its most acute form, and it gets its own lesson.

Next: scoping, and what the agent is allowed to reach.