How agents read a page

The obvious approach — send the page source — fails immediately on real sites. A modern page is often hundreds of kilobytes of markup, most of it framework wrappers, inline styles and analytics, with the actual content scattered thinly through it.

That is both unaffordable in tokens and actively harmful to accuracy, because the signal is buried. Every working browser agent therefore transforms the page into something much smaller before the model sees it.

The scale of the reduction is the point. A page that is three hundred kilobytes of markup frequently reduces to two or three kilobytes of structured description, and the smaller version is both cheaper and easier to reason about.

Browsers already build a structured description of a page for screen readers: what each element is, its label, its state, whether it is interactive. That tree is compact, semantically meaningful, and available through standard automation interfaces.

It is close to ideal for this purpose. A button appears as a button with its accessible name rather than as a nest of divs, which is exactly the abstraction a model needs to decide what to click.

There is a pleasing consequence here: sites built properly for screen readers are also the easiest for agents to operate. Accessibility work that was done for people turns out to be the thing that makes automation reliable, and sites that skipped it are hard for both.

• Element role — button, textbox, link, checkbox
• Accessible name — the label a screen reader would announce
• State — disabled, checked, expanded, required
• Structure — what contains what, so context is preserved

Some meaning is only visual: which option is highlighted, that an error appeared in red beside a field, that a button is visually disabled without being marked so, that a modal is covering the form.

For these a screenshot is sent instead of or alongside the tree. It costs considerably more per step, so most systems use structure by default and reach for the image when the structural view is ambiguous or an action unexpectedly failed.

The cost difference is large enough to change architecture. An agent that screenshots every step can be an order of magnitude more expensive than one that reads structure and falls back, which is why the hybrid approach is close to universal in production.

The common technique assigns each interactive element a numeric label and shows the model the labelled list, or a screenshot with those numbers drawn on it. The model then says "click 14" rather than describing a selector.

This is a deliberate design choice. Asking a model to produce a CSS selector invites it to invent a plausible one that matches nothing; constraining it to choose from an enumerated list of elements that genuinely exist eliminates that failure entirely.

It is the same principle as a dropdown rather than a free-text field. Removing the opportunity to produce something invalid is more reliable than validating afterwards, and it applies to model output as much as to user input.

Pages defeat this in predictable ways. Elements with no accessible name appear as an unlabelled button among six others. Content loads after the snapshot, so the model reasons about a page that no longer exists. Custom components built from divs do not report a role.

The practical mitigations are unglamorous: wait for the page to settle before snapshotting, re-snapshot after any action, and verify the expected change actually happened rather than assuming the click landed where intended.

The stale snapshot is the most common of these by a wide margin. Modern pages render progressively, and a snapshot taken a few hundred milliseconds early produces an element list that is already wrong by the time the model responds.

• Unlabelled controls — several identical "button" entries with no name
• Content that appears after the snapshot was taken
• Custom components that report no role or state
• Overlays and cookie banners intercepting the click
• Elements present in the tree but scrolled out of the viewport

Agents are becoming a category of visitor, and the things that make a page legible to them are the things that already made it legible to assistive technology. Semantic elements, real labels, accurate disabled and expanded states.

A page built from divs with click handlers is hostile to screen readers and to agents for exactly the same reason: nothing in it declares what anything is. That is now a commercial consideration as well as an accessibility one.

For a careers site in particular this matters. If candidates increasingly reach your application form through some kind of assistant, a form that no automated client can complete is a form that quietly loses applicants you never hear from.