The 5,750 tasks / 452 websites benchmark, and the layer it can't measure

What the two numbers actually encode

The headline figures are a deliberate jump in scope. WebVoyager, the benchmark Web Bench replaced, ran 643 tasks against 15 hand-picked sites. That was enough to rank agents, but narrow enough that an agent could be tuned to those 15 sites and look better than it was. Web Bench went the other direction: 452 websites sampled from the top 1,000 sites globally by traffic, spread across roughly 17 categories, with 5,750 tasks layered on top.

The tasks themselves split into two families. READ tasks ask the agent to navigate to a piece of information and pull it out. WRITE tasks ask it to change state: fill a form, log in, solve a 2FA challenge, download a file, create or update or delete a record. Across agents, READ scores sit well above WRITE scores, which is the single most useful thing the benchmark tells you. Reading the web is close to solved. Acting on it, reliably, is not.

That gap is the reason a number this large still leaves room for a product like Mediar. But the more important detail is hiding in the word "websites."

The 452 sites were sampled from public web traffic. That is the whole story.

Read the methodology and one line decides everything: the 452 sites were drawn from the top 1,000 websites globally by traffic. To enter that pool a system needs a URL and a measurable amount of public traffic. That is a perfectly reasonable way to build a browser-agent benchmark. It is also a filter that removes, by construction, every system most enterprise automation work is actually stuck on.

A SAP Business One window has no URL. A Jack Henry core-banking green-screen has no public traffic rank. An Epic chart, an Oracle EBS form, a mainframe terminal: none of them can appear in a ranking of websites because none of them are websites. So the desktop line-of-business surface is not under-represented in Web Bench. It is absent. It was never eligible.

This is why Mediar does not drive a DOM. It reads what an application exposes through the operating system's accessibility APIs, the same interfaces a screen reader uses. A window that has no web address still has an accessibility tree, and that tree is what the agent locates targets in. Here is the difference made concrete, using the open-source Terminator engine inspecting a SAP window:

There is no headless endpoint and no page to render. The targets are accessibility nodes, not DOM elements. A browser-agent benchmark has no way to express this surface, which is precisely why a high score on one says nothing about it.

What a web benchmark covers vs where the desktop work lives

This is not a knock on Web Bench. It measures what it set out to measure, and it does it at real scale. The point is to read the score for what it is, then look at the column it cannot reach.

None of these could appear in the 452

Every system below is a place real teams run hundreds of repetitive workflows a week. Not one of them has a URL or a public traffic rank, so not one of them is eligible for a benchmark built from web traffic. This is the surface Mediar was built for.

The honest counterargument: isn't everything moving to the browser?

Yes, new software ships as SaaS, and for those workflows a browser agent measured by Web Bench is genuinely the better fit. We say this plainly: if your data lives in a modern web app, you do not need an accessibility-API approach, and a browser agent will serve you well.

But the systems of record in banking, insurance, healthcare, and manufacturing are not migrating off SAP, Oracle, Jack Henry, or Epic this decade. Those platforms were installed over twenty years and carry regulatory and integration weight that makes replacement a multi-year program, not a quarter. The desktop layer is not a shrinking legacy footnote. It is where the unautomated work has collected precisely because browser tools cannot reach it. A benchmark that grows from 15 to 452 websites is real progress on one axis and zero progress on that one.

How to read the benchmark before you buy

Use Web Bench for what it is good for. A high score is solid evidence an agent can navigate and act on public websites, and the READ-versus-WRITE gap is an honest signal about how far reliable web acting still has to go. When a vendor cites it, that is a fair claim about browser work.

Then ask the question the benchmark cannot answer: where is my stuck work actually running? If the answer is a SAP transaction, a green-screen, an EHR chart, or a mainframe form, the right test is not a website score. It is whether the agent can watch that desktop workflow once and then execute it through the accessibility tree, self-healing when a label or layout shifts, with audit logs the compliance team accepts. That is the test Mediar is built to pass, and it is the one a 452-website benchmark was never designed to run. The engine behind it is open source at github.com/mediar-ai/terminator.

Frequently asked