Skyvern’s WebVoyager benchmark, line by line: 85.85%, 15 sites, and the surface the benchmark does not measure.

The 15 sites WebVoyager actually covers, with task counts.

The next table is what most coverage of Skyvern’s WebVoyager number leaves out. The 643 figure rolls up to a per-site distribution that is easy to count yourself. Curl the file at raw.githubusercontent.com/MinorJerry/WebVoyager/main/data/WebVoyager_data.jsonl and group by web_name. Counted on 2026-05-08, the per-site totals are below.

Two facts worth pulling out before moving on. First, the distribution is roughly flat: every site contributes between 41 and 46 tasks. There is no single site dominating the 643. Second, every one of the 15 is a public consumer-facing web property reachable from a stock Chromium browser. There is no site on the list that requires a corporate VPN, a Citrix client, or an installed Win32 thick-client. That second fact is the load-bearing one for how to read Skyvern’s 85.85%.

Where Skyvern sits on the public leaderboard.

The Steel.dev leaderboard at leaderboard.steel.dev collects published WebVoyager scores for every major web-agent framework. Snapshot below was pulled on 30 April 2026. The leaderboard itself notes that “rows may use different evaluation settings and are not always strict apples-to-apples,” which is honest: most of the variance between adjacent rows is grader configuration, dataset patches, and time-sensitive task drift, not a fundamental capability gap.

Two rows on this leaderboard deserve a footnote in any honest read. Anthropic’s Computer Use 3.5 row at 56.0% is sampled on 50 of 602 tasks, not the full set; the headline number does not translate one-for-one to a full-suite run. The original WebVoyager baseline at 59.1% is the 2024 academic-paper number and uses the un-patched dataset, including the time-sensitive Booking and Google Flights tasks that have since drifted.

The loop the benchmark scores: plan, act, validate.

The way Skyvern 2.0 went from roughly 45% to 85.85% on WebVoyager is architecturally simple to describe. The framework adds a planner and a validator on either side of the actor agent. The planner decides what sub-step to attempt; the actor fires the click; the validator re-reads the post-action screenshot and decides whether the page actually went where the planner expected. If not, control loops back to the planner with the error context. WebVoyager’s GPT-4V grader watches the entire trajectory and either passes or fails the task at the end.

Two things follow from this loop being the thing that gets graded. One, every step happens against a rendered Chromium viewport. The planner reads the DOM and screenshot, the actor selects a labeled DOM node, the validator reads a screenshot of the same Chromium tab. Two, the validator is what made the 17-point jump from 1.0 to 2.0 possible: a single-agent loop has no way to know it just took a wrong step, so any error compounds; a separate validator can catch and re-plan around it. The validator is also why the framework feels solid on portals it has not seen before.

What WebVoyager is and is not measuring.

A benchmark’s honest contract is “the score I publish carries weight on the surface I tested.” WebVoyager tested 15 public consumer-web sites with an automated GPT-4V grader. The score earns weight on that surface. It earns no weight on a surface the benchmark did not see. The next table is the boundary, side by side. The left column is what WebVoyager does cover (and where Skyvern’s 85.85% is real evidence). The right column is the enterprise legacy desktop work where Mediar lives and where the benchmark is silent by construction.

The boundary is sharp. A SAP GUI window is not a Chromium tab. An Oracle Forms session is not a tab. A Jack Henry green-screen terminal is not a tab. An Epic Hyperspace patient chart inside a Citrix shell renders no DOM the agent can read. None of those surfaces is in WebVoyager, and none of them is in any other browser-agent benchmark either, because the input layer the benchmarks assume (a Chromium-rendered viewport with a stable DOM) is the input layer those surfaces are missing.

How a CFO should read Skyvern’s 85.85% before signing the RPA-replacement contract.

Two questions, in this order. First: of the workflow hours per week your team is trying to take off the floor, what percentage lives inside a Chromium tab on a public or near-public consumer web property? Vendor portal logins, lead enrichment, public-site downloads, payer claim-status checks on a hardened extranet, the long tail of B2B SaaS form-fills. That percentage is the slice of your portfolio where the WebVoyager number transfers cleanly. The Skyvern launch post is not exaggerating: a Chromium-tab agent that scores 85.85% on a 643-task suite of consumer-web tasks is a real piece of capability for that slice.

Second: of the same hours per week, what percentage lives on a Win32 thick-client, a Citrix-shelled enterprise app, a mainframe terminal, or a desktop spreadsheet the user is editing in place? That percentage is the slice where the WebVoyager number says nothing useful. There is no row in the 643 tasks that exercised this surface, no architectural transfer is implied, and no Skyvern tier currently ships a Windows desktop runtime sized for it. A buyer who reads the headline and assumes the score generalizes is quietly importing a benchmark scope mismatch into the contract.

The honest answer for most enterprise teams running existing UiPath or Power Automate workloads is a portfolio split, not a single pick. Browser-tab work goes to the agent that is strong on browser tabs. Desktop work goes to the agent that has a desktop runtime. Mediar’s side of that split prices in wall-clock minutes on Windows ($0.75 per minute of agent time, drawn against a $10,000 turn-key prepay that converts to credits with a small bonus) and drives the desktop through the Windows accessibility tree, the same interface screen readers use. That is a different unit on a different surface. Both can be true on the same purchase order.

Frequently asked questions