WebBench: 2,454 tasks, 452 websites, and the layer the benchmark cannot score

What the 2,454 tasks actually contain

WebBench grew out of WebVoyager (15 sites, 642 tasks). Halluminate widened coverage to 452 sites across 17 traffic-ranked categories, generated roughly 5,750 candidate tasks, then trimmed to 2,454 after dropping tasks that broke when the live sites changed. Every task is tagged by what it asks the agent to do. The split is the whole story:

Read is nearly two-thirds of the set. The four action categories, collapsed together, are the other 35.6% (874 tasks). That ratio is not an accident of sampling: write tasks are simply harder to author, harder to verify, and far harder for an agent to complete. Which is the first thing the scores confirm.

Reads are basically solved. Writes are not.

On read tasks, 5 of the 7 agents Halluminate tested cleared 70%, and most landed above 75%. If your job is “find this fact on this page,” a 2026 browser agent does it. The headline 66.0% overall score for the best fully automated agent (Anthropic CUA) is carried mostly by that read-heavy majority.

Read that again with an operations hat on. The tasks an enterprise actually pays people to do all day are write tasks: post the invoice, key the claim, update the customer record, pull the statement. WebBench puts a hard number on how well agents do those today, on the public web where they at least have a DOM and a URL to hold onto: fewer than half. Logging in, form-filling, and downloads are named explicitly as the weak spots.

The constraint nobody writes down: every target has a URL

Here is the part the other write-ups skip. WebBench samples from the global top-1000 websites by traffic. That selection rule decides, in advance, what the benchmark can ever say something about. It can score an agent on Amazon, on a SaaS dashboard, on a government portal. It cannot score an agent on the systems where a community bank, a regional carrier, or a hospital actually runs its back office, because those are not websites.

This is not a knock on WebBench. It is a careful, useful benchmark for exactly what it claims to measure. The mistake is reading a browser-agent score and concluding something about enterprise automation in general. The hardest, highest-value back-office work sits on desktop apps with no API and no browser surface, and no public benchmark of websites will ever tell you how an agent does there.

Where the 46.6% problem and the no-URL problem both go away

A browser agent struggles with write tasks partly because it is reverse-engineering intent from a rendered DOM and, when that fails, from pixels. On a legacy desktop app there is no DOM to reverse-engineer at all. Mediar takes a different input entirely: the OS-level accessibility tree, the same structured interface a screen reader consumes. Controls have stable roles and names whether or not the app ever shipped a web front end.

That is the open-source Terminator SDK (github.com/mediar-ai/terminator, MIT, Windows-only) underneath. Because the agent locates a field by its accessibility role and name instead of a pixel position or a brittle selector, a write task on SAP GUI is the same shape of operation as a write task in any other Windows app: find the control, set the value. It also self-heals when a label or layout moves, because there is no coordinate or selector to break.

We have watched this play out in production, not in a benchmark. An F&B chain moved off UiPath to Mediar and their CFO told the board they are saving 70% on costs. At one mid-market insurance carrier, claims intake went from 30 minutes per claim to 2 minutes, which is about $750K a year on their own headcount math. Those are write tasks, on legacy desktop systems, the exact intersection WebBench is structurally unable to put a number on.

WebBench, answered