Browser AI, explained

Browser AI is a model driving a real web browser: reading the page, deciding what to do, clicking, typing, scrolling, and reading what happened next. It operates the interface built for humans rather than an interface built for programs.

That is the entire distinction from conventional automation. A scraper is told where things are; a browser agent works out where things are. One breaks when the page changes; the other adapts, at considerable cost.

The framing that helps is to think of it as hiring a temp rather than writing a macro. A macro does exactly what you specified and nothing else; a temp works out what you meant, occasionally gets it wrong, and costs considerably more per task.

A traditional script encodes a path: find this selector, click it, read that element. It is fast, cheap and completely deterministic — and it breaks the moment a class name changes.

A browser agent is given a goal. It looks at the page, reasons about which element probably does what it needs, and tries. When the layout changes it usually still works, because it was never depending on the layout in the first place.

The failure modes are opposite, and that difference matters more than the cost. A scraper stops when it cannot find something, which is loud and obvious; an agent picks the nearest plausible element and continues, which is quiet and sometimes wrong.

• Scraper: fast, cheap, deterministic, brittle to any change
• Browser agent: slow, expensive, adaptive, non-deterministic
• Scraper: fails loudly when a selector is missing
• Browser agent: may quietly do the wrong thing instead of failing

There are two approaches and they trade off sharply. One feeds a processed version of the page structure — the accessibility tree or a simplified DOM — which is compact and cheap but loses visual meaning. The other feeds a screenshot, which captures what a human sees but costs far more tokens per step.

Most production systems use the structural approach with screenshots as a fallback for pages where layout carries meaning. Understanding this trade-off explains most of the cost and latency differences between tools.

A pleasing side effect is that sites built properly for screen readers are the easiest for agents to operate. The accessibility tree is exactly what a browser already produces for assistive technology, so accessibility work done for people turns out to be what makes automation reliable.

Every step is a model call. Look at page, decide, act, observe result — and a form with eight fields is potentially eight or more round trips, each costing tokens and seconds.

This is why browser agents are reserved for cases where nothing better exists. If there is an API, use the API. It will be a hundred times faster, cost almost nothing, and fail in ways you can predict.

The cost also compounds with verification, which is not optional. An agent that does not re-read the page after each action cannot tell whether the click landed, and the verification roughly doubles the number of calls — which is worth paying and worth budgeting for.

A browser agent reads pages written by other people and can act on what it finds. That combination is the precondition for indirect prompt injection: text on a page can address the agent rather than the reader, and the agent has no reliable way to tell the difference.

The mitigation is structural rather than a better instruction. Keep the component that reads pages separate from the component that can submit, restrict which hosts it can reach, and require a person to confirm anything irreversible.

This is also why "it can do anything a human can" is a warning rather than a feature. The useful version of a browser agent is one whose capabilities have been deliberately narrowed to the task, so that being persuaded gets it nowhere.

The failures are consistent enough to list, and almost none of them are the model failing to reason. Most are timing: the agent captured the page a few hundred milliseconds before it finished rendering, so it is deciding about a layout that no longer exists by the time it acts.

The rest are representation problems. A control built from divs reports no role, so it appears as an unlabelled element among six others. A cookie banner or modal intercepts the click. An element exists in the page structure but is scrolled out of view and cannot actually be clicked.

The mitigations are unglamorous and effective: wait for the page to settle before capturing, re-capture after every action, and verify that the expected change occurred rather than assuming the click landed. That roughly doubles the interactions, which is the cost of not being confidently wrong.

• A stale capture taken before the page finished rendering
• Custom components that report no role or accessible name
• Overlays and consent banners intercepting the click
• Elements present in the structure but outside the viewport
• Content that loads only after a scroll or an interaction

The genuine use case is the long tail: a task spread across many sites that each do things slightly differently, where writing and maintaining a script per site would cost more than the automation saves.

Job applications are close to the ideal example. Thousands of employers, several applicant tracking systems, endless variation in form fields, and no useful API for most of them. Writing a scraper per employer is impossible; a system that can read a form and work out what it is asking is the only approach that scales.

Even there, the economics improve with memory. Learning a site’s form layout once and replaying it deterministically afterwards — falling back to reasoning only when the page changes — turns an expensive approach into an affordable one over repeated use.