CUA AI: what a computer-using agent is, and why the term hides two very different products

Why the term feels new and is not

The acronym CUA showed up in the popular vocabulary when OpenAI published Operator in January 2025, and most of the explainers published since then trace the lineage back to that launch. The underlying idea is older. Anything that drives a graphical interface using something other than an API qualifies. UiPath was a CUA before the abbreviation existed. So was AppleScript, AutoIt, the Selenium IDE recorder, and every screen-scraping tool the bank operations team was running in 2008. What changed in 2024-2025 is that vision-language models got good enough to substitute for the part of those systems that a developer used to do by hand: read a screen, decide what control to touch, and emit the touch.

That substitution is the actual claim. The marketing version of CUA collapses it into a one-liner: "the model uses your computer." The accurate version is more useful. It says the model has been promoted from the design loop into the perception-action loop, and lets you ask the next question. Promoted to where, exactly? On every step? On some steps? Once at the start, never again? Each answer corresponds to a different product with different cost, latency, and audit properties.

Most pages on this topic do not ask that question. So the rest of this one is about it.

The two architectures, side by side

The split is not about which model the agent uses. GPT-4o, Claude 3.7, Gemini 2.5, and UI-TARS all show up on both sides. The split is about where the model runs in the lifecycle of a workflow.

A short detour into benchmarks

The number a buyer should ask about first is the OSWorld success rate. OSWorld is the benchmark that matters for this category because it tests full computer-use tasks against real desktop environments rather than scripted web-only flows. OpenAI's CUA was the first agent to push past the previous state of the art on it.

Six tasks in ten fail. That is a research result, and a real one, and worth celebrating in the context of where the field was a year earlier. It is also not a number you can stage a hundred-step claims-intake flow on. A workflow's success probability is the product of its step probabilities, so a hundred steps at even 95% per-step accuracy comes out to a coin flip across the whole run. OSWorld at 38% across the whole task means that on a hundred-step flow you are running effectively zero percent end-to-end without retries.

A June 2025 paper called OSWorld-Human adds the time picture. A task a human finishes in two minutes can take an agent over twenty, with 75 to 94 percent of that time spent in the LLM's planning and reflection calls. The cost picture tracks the time picture: every UI step is at least one inference call, sometimes several.

The vision-loop architecture pays this tax by design. The tree-based architecture does not, because the model is not in the loop. The trade is that you give up open-ended in-the-moment reasoning at runtime in exchange for a deterministic replay of a recorded workflow. Whether that trade is good depends on what the workflow is.

Inside a tree-based CUA: dual-channel input fusion

Here is one implementation detail that does not appear in any of the popular CUA explainers, and that turns out to matter for any agent that has to handle a real enterprise mix of native windows and embedded browsers.

When a user clicks a control inside a Chromium browser, two events fire in parallel. A Chrome extension running in the page publishes a BrowserClick event carrying the CSS selector for the element. Independently, the Windows UI Automation layer publishes a Click event carrying the accessibility-tree selector for the same element. The same physical click produces two structured records of itself, viewed from two different layers of the stack.

A vision-only CUA never sees either of these. It only sees the screenshot. A tree-based CUA can take both, fuse them inside a short time window, and emit a single run_command that carries both selectors. At replay time, the runtime tries the higher-fidelity CSS selector first; if the page is rendered through Internet Explorer compatibility mode, an iframe boundary, or any other surface where the DOM is not addressable, it falls back to the UI Automation selector. The fusion logic is in apps/desktop/src-tauri/src/workflow_recorder.rs lines 176 to 194, and the runtime fallback is in apps/desktop/src-tauri/src/mcp_converter.rs starting at line 1034, both in the open-source repo.

That single design choice is why the same agent that drives a SAP GUI window can also drive a hybrid app where some surfaces are native Win32 and others are embedded Chromium. Each surface gets the most precise selector available; neither needs a vision model. Pixel-vision agents cannot do this because they have discarded the structured input before they ever reach the model.

The runtime grep is the architecture audit

You can disambiguate the two architectures with a one-line test on the runtime executable. If the runtime imports an LLM SDK, the model is in the loop. If it does not, the model has been confined to authoring time.

For Mediar, the runtime executable is one Rust file at crates/executor/src/services/typescript_executor.rs. It is 871 lines. Its job is mechanical: pull a workflow off a Postgres queue, build the MCP arguments (workflow URL, secrets, trace id), and call the MCP execute_sequence tool against the target Windows session. The model only enters the picture in a completely separate file at authoring time, in apps/desktop/src-tauri/src/recording_processor.rs's analyze_step function around line 1197, where Gemini Pro reads the captured tree once for each recorded step.

You can run the test yourself. Clone github.com/mediar-ai/terminator, open the executor file, and grep for openai, claude, anthropic, or gemini. You will find zero matches. That is the architectural commitment in source form: the model is allowed to think slowly about one workflow at a time during recording and is not allowed to be in the loop when the workflow is replaying against a customer's general ledger at 4 a.m.

The same grep against a vision-loop CUA finds the LLM client import in the runtime, because that is where the architecture puts it. Neither answer is wrong; they are different products. The grep just makes the choice legible.

When each architecture is the right call

The vision-loop architecture is the right default when the workload is exploratory or one-shot, when the surface genuinely has no structured channel (a third-party PDF report you cannot script, a CAPTCHA-laden site, a video frame), and when the team has the budget for per-step inference. Operator booking a flight, a researcher asking Claude Computer Use to summarize a Notion page, or a developer prototyping with trycua/cua all fit. The per-step model call is a feature in those settings because the system needs to react to whatever shows up on screen.

The tree-based architecture is the right default when the workload is repetitive, the target surface publishes an accessibility tree (which covers SAP GUI, Oracle EBS, Jack Henry, Fiserv, FIS, Epic, Cerner, eClinicalWorks, mainframe terminal emulators, and most of the Windows desktop estate), and the team needs deterministic replay for audit. A claims intake flow at a mid-market carrier, a customer onboarding flow at a regional bank, a patient verification flow in a clinic, and a master data update on SAP B1 all fit.

The honest summary is that the two architectures address adjacent but different problems. The vision-loop CUA is the right answer for novel exploration; the tree-based CUA is the right answer for unattended replay. A team buying for the second and shown a demo of the first will be disappointed by the OSWorld score in production. A team buying for the first and handed a tree-based runtime will find it inflexible. The confusion comes from the shared name.

What this means if you are evaluating a CUA

The first question to ask a vendor is not which model they use. The first question is where the model runs. If the answer is "on every step from a screenshot," you have a vision-loop product. Plan for the per-step latency, the per-run token bill, and the audit conversation about why the action sequence is non-deterministic. If the answer is "once at recording time, the runtime is plain code reading the OS accessibility tree, and the workflow file lives in git," you have a tree-based product. The bill is bounded by runtime minutes, the audit artifact is a diffable code file, and a UI rev does not require regenerating the workflow under a fresh model session.

The second question is what the runtime executable looks like. Ask to see it. If the vendor cannot point at a single file (or will not), the architecture probably does not support the determinism story they are telling.