Legacy desktop systems already have an API. It is the accessibility tree.

The pixel-first default is a category error

Read the current top results for this question and you get a consistent story: legacy systems have no API, vision-language models can read screens, therefore point a VLM at the screen and have it click. Microsoft Copilot Studio's Computer Use, Bytebot, OpenAI's computer-use agents, and most of the “agentic automation” pitches all sit on this premise. The premise is missing one fact.

Legacy desktop systems do have a machine-readable surface. The OS attaches one because screen readers need it. JAWS and Narrator do not look at the pixels of an SAP transaction code; they ask the window for its tree of controls and read it aloud. That tree contains the role of every node (Window, Button, Edit, ComboBox, CheckBox, Tree, TreeItem), the human-readable name (the same label a sighted user reads), and a small set of attributes that capture state (value, checked, selected). A 30-year-old Win32 form published the same tree the day it was compiled, because the accessibility surface lives at the OS layer, not in the application code.

A pixel-first agent ignores all of this. It sends a screenshot to a model and pays for inference to recover information that the OS would have handed it for free.

What the model actually reads from a SAP GUI window

The Mediar recorder serializes a captured window into a flat, indented list, one node per line, in this exact format. The serializer is generate_simplified_ui_tree_string at apps/desktop/src-tauri/src/recording_processor.rs:1014. A real Customer Master Data window from SAP comes out looking like this:

1.  I.    [Window] 'Customer Master Data Maintenance'
2.   II.  [Toolbar] 'Standard'
3.    III. [Button] 'Save' {focusable=true}
4.    III. [Button] 'Display'
5.   II.  [Group] 'General Data'
6.    III. [Edit] 'Customer' {value="0000470192"}
7.    III. [Edit] 'Title' {value="Mr."}
8.    III. [Edit] 'Name 1' {value="Imperial Treasure Pte Ltd"}
9.    III. [Edit] 'Search Term' {value=""}
10.   III. [Edit] 'Country' {value="SG"}
11.   III. [Edit] 'Postal Code' {value="048619"}
12.   III. [ComboBox] 'Reconciliation Account' {selected=true}
13.   III. [CheckBox] 'Posting Block' {checked=false}

That is what gets fed to the model at authoring time. No image, no OCR, no pixel coordinates. A SAP transaction with 80 controls becomes about 80 lines of text. The model reads the tree, decides which nodes the workflow touched in which order, and writes a TypeScript file that targets each node by role and name. The authored file is what runs in production.

The four-strategy fallback when the UI shifts

The classic complaint about RPA is that selectors break the moment someone renames a label or reorders a panel. The classic complaint about vision agents is that they hallucinate when the screenshot looks different. The tree-based approach gets a third option: when the recorded element does not match cleanly, walk a cascade. The cascade in apps/desktop/src-tauri/src/focus_state.rs lines 168 to 196 has four steps:

1. Strategy 1

   Match by automation or accessibility ID

   Stable across most UI changes because it is the property the OS uses to identify the control to a screen reader. Survives label rewrites, panel reorders, and theme changes.

2. Strategy 2

   Match by window plus bounds

   If the ID has changed (a new SAP support pack can do this), find the window, walk the tree, pick the node whose rectangle is closest to the recorded one. Survives most layout-stable patches.

3. Strategy 3

   Match by visible text

   If the rectangle moved too, fall back to the human-readable name. The label on the field is what the analyst recorded against, so this is the closest analog to what they would do if asked to find the field again themselves.

4. Strategy 4

   Window-only focus

   If none of the above land, focus the parent window so the next event in the workflow has a chance to fire against the right surface. The run is flagged for review, not killed silently.

A SAP support pack that nudges a field down by 12 pixels triggers nothing because the id matches. A pack that renames the field label triggers strategy two. A pack that does both still gets caught by visible-text matching. A pack that breaks all four flags the run rather than silently typing into the wrong field, which is the conservative default for anything posting to a general ledger.

The runtime is the proof

A claim that the model only runs at authoring time is testable. Read the runtime. The Mediar production executor is a single Rust file at crates/executor/src/services/typescript_executor.rs. It is 871 lines. Its job is mechanical: pull a workflow off a Postgres queue, build MCP arguments (file URL, secrets, trace id), and call the MCP execute_sequence tool against a Windows session. The MCP server wraps the Terminator SDK, which is the part that talks to UI Automation.

Grep that file for gemini, openai, claude, or anthropic and you find zero matches. That is the architectural commitment. The model is in the recorder pass, where it has time and only sees one workflow once. The runtime that types into a customer's SAP system at 4am is plain code reading the tree.

When the tree does not have what you need

The accessibility surface is rich enough for almost every Windows enterprise app, but it is not magic. Two known gaps. First, vintage custom-paint apps that bypass standard controls and render directly to a device context. These are rare in regulated workloads because they break screen readers too, and most enterprises retired them in the last accessibility audit cycle, but a few survive in manufacturing and trading desks. Second, partially-bridged Java AWT clients where some controls publish to UIA and some do not.

For both, the right answer is a hybrid. Read the tree where it is rich, fall back to OCR plus a small vision call for the residual controls. The hybrid pays for inference only on the controls that earn it, which is a couple of cents per run rather than a model call per UI step. The same hybrid pattern appears for browser-rendered legacy apps: when the target is Internet Explorer or Edge in compatibility mode, Mediar prefers DOM capture via a Chrome extension, then falls back to UIA. The handler that combines them is in apps/desktop/src-tauri/src/mcp_converter.rs starting at line 1034.

The point is not that the tree wins everywhere. It is that the tree wins so often, and the cost of reading it is so close to zero, that it should be the default and pixels should be the fallback, not the other way around.

What this means in practice

If you are evaluating an AI agent for a workload that lives in a Windows desktop system without an API (and that covers most of SAP GUI, Oracle EBS, Jack Henry, Fiserv, FIS, Epic, Cerner, eClinicalWorks, mainframe terminal emulators, and the long tail of internal Windows apps), one question separates the production-ready tools from the demo-ready ones: where does the model live in your execution architecture.

If the answer is “the model decides the next action on every step from a screenshot,” you have bought a vision-based agent. Real products work. Plan for the per-step latency and per-run token cost in your ROI math, and plan for the audit conversation where you explain why the action sequence is non-deterministic.

If the answer is “the model runs once at authoring time, the runtime is plain code reading the OS accessibility tree, and the workflow file lives in git,” you have bought a tree-based agent. The bill is bounded by runtime minutes, the audit trail is a code file you can diff, and a UI rev does not require regenerating the workflow with a fresh model session.