RPA, legacy desktop, accessibility agents: the wire format

An agent that watches a workflow once, then replays it by emitting structured tool calls (click_element, type_into_element, press_key_global, activate_element, scroll_element, navigate_browser, execute_browser_script, run_command) against the OS accessibility tree, not against pixels and not against fragile control-tree paths. On Windows the surface is UI Automation (UIA), the same surface NVDA, JAWS, and Narrator use to read SAP, Oracle, Jack Henry, and Epic to a blind user. The agent reads each control by role and name, attempts the primary selector, walks a short fallback cascade if the UI moved, and only fails the run if every fallback misses.

UiPath, Automation Anywhere, and Blue Prism record a stored path through the control tree for every click. The path includes element indices, automation ids, and class names that were correct at recording time. The path breaks every time the UI shifts: a SAP support pack renumbers a control, a WinForms recompile churns ids, a theme change repaints a panel. An accessibility-agent does not store a path. It stores the role and name (role:Edit && text:Customer), and resolves fresh on every run. The fallback cascade is part of the tool argument (fallback_selectors), not a separate maintenance job.

Vision and pixel approaches read the rendered screen and infer where to click. They clear the surface bar (an LLM can see a SAP window) but stall on three things: latency (30-60 seconds per UI step from a model call), cost (tokens per step times steps per workflow), and determinism (the model can pick a different button on the second run). An accessibility-API agent reads a structured tree of role/name/value/state directly from the OS in microseconds, with no inference call, and the runtime that replays the workflow contains no LLM calls at all. The model is in the recorder, not in the hot path.

Three building blocks. role:X targets a UIA control type (Edit, Button, ComboBox, Window, Pane, ListItem, Tab, Document). role:X && text:Y narrows by the human-readable name the form definition published to UIA. The >> operator scopes the search: role:Window && text:Customer Master >> role:Edit && text:Customer finds the Customer field inside the Customer Master window, not in some other modal that happens to have the same field. Every selector compiles down to a UIA TreeWalker call against the live process; nothing is stored as a coordinate.

Every tool call carries a primary selector and a comma-separated list of fallback selectors. The default ceiling is three fallbacks (max_fallback_strategies = 3 in the converter's default config). If the primary misses, the runtime tries each fallback in order. Typical cascade: scoped selector (window >> element) first, unscoped element second, role-only third. Most quarterly SAP support packs, WinForms rebuilds, and Epic upgrades are absorbed by one of the first two fallbacks. A change that breaks all three flags the run for human review instead of silently posting to the wrong field, which is the conservative default for anything touching a general ledger or a clinical chart.

As a TypeScript file in a git repo. The recorder converts each captured event into an MCP tool step, the model converts that sequence into a reviewable .ts file, and the executor reads the .ts file at runtime. The model runs once, at authoring time, on a screen-share with the analyst who owns the workflow. The runtime is a Rust crate (see github.com/mediar-ai/terminator under MIT). Grep the runtime for openai, claude, or anthropic and you find zero matches. The architectural commitment is: the model is in the recorder, not in the hot path that types into your general ledger at 4am.

Anything that publishes a UIA tree on Windows. That includes the systems where UiPath stalls and where browser agents cannot reach at all: SAP GUI, SAP Business One, Oracle EBS, Jack Henry SilverLake and Symitar, Fiserv DNA and Premier, FIS IBS, Epic Hyperspace, Cerner (now Oracle Health), eClinicalWorks, MEDITECH, Citrix-published thick clients, VB6 and Delphi internal apps, MFC and WinForms apps, mainframe terminal emulators (Reflection, BlueZone, PuTTY). The reach is identical to whatever a screen reader can read aloud, which is the regulatory bar Windows enforces.

Yes. The runtime that resolves selectors against UIA, walks fallbacks, and dispatches clicks is published under MIT at github.com/mediar-ai/terminator. The MCP server, the selector parser, and the Rust crate are in the same repo. The recorder and the no-code web app at app.mediar.ai/web sit on top of it. Teams that want to extend the agent (a custom selector primitive, a process-specific verifier, a new MCP tool) work in the same SDK.

$0.75 per minute of runtime, billed against credits. The $10,000 turn-key program fee converts to credits with a bonus, so it is effectively prepaid usage. No per-seat licensing, no developer certification, no $250K annual maintenance line for selector cascades. Published deployments: an LG-supplier F&B chain reported a 70% reduction vs UiPath to their board, a mid-market insurance carrier saved roughly $750K a year on claims intake (30 minutes per claim down to 2), a community bank compressed customer onboarding from 8 weeks to 2 weeks on Jack Henry. The numbers are workflow-specific; we will price yours per minute on a screen-share before quoting.