Grounding the Action

In one line: the model has decided to click the search button, and something has to turn that intention into an actual element on an actual page.

Three ways to point at something

Coordinates. The model outputs a pixel position and the browser clicks there. Direct, and it requires the model to localise precisely from an image — a task models are mediocre at, and one where a small error hits the wrong element rather than failing cleanly.

A selector. The model writes a CSS or XPath expression. Precise when correct, and it asks the model to author a query against markup it can only partially see, producing selectors that match nothing, match several things, or match the wrong thing.

A label. The observation numbers every interactive element, and the model outputs a number. The system already knows exactly which element that is.

The third is the one to build, and it is the decision that most determines whether a browsing agent works.

Set-of-Mark

The technique, and it is worth being able to describe precisely.

Enumerate the interactive elements. Draw a numbered box around each one on the screenshot, and emit the same numbers in the accessibility tree listing. The model sees a picture with labelled boxes and a list saying what each number is, and it replies with a number.

The property that makes it work: the two representations are aligned. Without it, the model sees a screenshot and a separate text listing and has to match them itself — which is the localisation problem again, just moved. Drawing the boxes using coordinates taken from the accessibility tree is what makes label 14 in the picture and label 14 in the list provably the same element.

That alignment is the mechanism. Everything else is bookkeeping.

What it fixes

Three failure modes disappear.

Ambiguity. "Click Submit" on a page with three Submit buttons is unresolvable; "click 14" is not.

Localisation error. No pixel estimation, so no near-misses.

Invalid selectors. The model is choosing from a menu rather than composing a query, so it cannot produce something that matches nothing.

And a fourth, subtler one: the label set is the affordance list. The model can only act on elements the system enumerated, which means it cannot invent an interaction that does not exist — a meaningful constraint on an agent's ability to go somewhere unexpected.

What it does not fix

Being honest about the residue.

Labels are unstable across steps. The page re-renders and element 14 is now something else. So a label is valid only for the observation that produced it, and the agent must never carry a label between steps. That is a real bug people write.

Enumeration can miss things. An element with no accessible role, a custom control built from divs, something rendered in a canvas. If it is not in the tree, it gets no label, and the agent cannot act on it at all.

The label says what, not why. "Button, 14" does not say whether clicking it submits, resets or opens a dialog. The accessible name usually helps and does not always.

The mitigation for the second one is the interesting part: fall back to coordinates for elements that exist visually but not semantically, and treat it as a degraded mode with lower confidence rather than a normal path.

The action vocabulary

Small and closed, which is a design choice worth defending.

Click a label. Type text into a label. Select an option. Scroll. Navigate to a URL. Go back. Wait. And a few meta-actions: expand this region, read this section fully, finish with an answer.

Keeping the set small has three benefits. The model is choosing from a menu rather than composing arbitrary automation, which makes valid output far more likely. Every action is implementable reliably. And the agent's possible behaviours are enumerable, which is what makes the system reviewable at all.

The temptation to add "execute this JavaScript" should be resisted in almost every case. It collapses the action space into arbitrary code, makes behaviour unreviewable, and hands an injected instruction a general-purpose capability. If a task genuinely needs it, that is a signal the task wanted an API.

Typing is not one action

A detail that causes real bugs.

Filling a field is often click, clear, type, and sometimes a keypress to confirm or dismiss an autocomplete. An agent that models this as "type into 14" will leave text appended to an existing value, or leave a suggestion dropdown open that swallows the next click.

So the type action should be explicit about clearing, and about what follows — whether to press Enter, press Escape, or click elsewhere to dismiss. Those are different intentions and conflating them produces a class of failure that is confusing to debug because the typing itself worked.

Next: acting on a page that is still moving.