Workflow assistant: the test isn’t whether it can click and type

Where the term has drifted

The phrase "workflow assistant" now covers three different products in three different layers, and most pages on the subject blur the distinction. The blur is fine for a shopping comparison and bad for an architecture decision, so it is worth pulling apart up front.

The first layer is the API-plumbing assistant. Zapier, Make, n8n, Gumloop, and most of the platforms that surface in "AI workflow tools 2026" roundups live here. You describe a trigger, a condition, and an action; the platform translates that into HTTP calls against a fixed catalog of integrations. The assistant part is usually a chat that drafts the configuration. The runtime is just an iPaaS.

The second layer is the screen-watching assistant. Adobe Firefly, Microsoft Copilot inside Office, the Operator-style browser agents, and the new wave of vision-based desktop agents like Simular and Skywork live here. They look at the screen (often as pixels through a vision model) and try to act on what they see. They are often impressive in a demo and brittle in a production loop, because the perception layer is non-determinist and the recordings are not durable artifacts.

The third layer is the recording-and-replay assistant. Mediar lives here, and so does any RPA platform that read the OS accessibility tree honestly (UiPath at its best, parts of Power Automate Desktop, Blue Prism). The user does the task once. The assistant captures structured events and the accessibility tree. Output is a workflow file that can be re-run on a schedule or on demand. This is the layer where "workflow assistant" stops being a chat veneer and starts being something you can deploy.

Inside the recording loop

The rest of this page documents the recording-and-replay layer using Mediar’s desktop agent as the worked example. The source files cited live in the apps/desktop/src-tauri/src/ directory of the Mediar product monorepo.

The six events the assistant pays attention to

The first thing the assistant decides is what not to think about. A live screen produces hundreds of low-level events per second: mouse moves, focus changes, scroll wheels, hovers, property updates inside the accessibility tree. Treat them all as candidates for a step and the LLM bill alone makes the assistant unaffordable, never mind the noise in the labels.

Mediar narrows this firehose at one named function:

fn is_meaningful_event_type(event_type: &str) -> bool {} at recording_processor.rs:250

Six is a small number on purpose. Each one is an action the user deliberately took: pressing a button, clicking inside a browser, finishing a text input, switching tabs, switching applications, opening a file. Hovers, scrolls, and focus events get captured for replay accuracy, but they never trigger a Gemini call. This is the difference between $0.75/minute of runtime and a bill that scales with cursor movement.

The four stages, in the order they actually run

Stage 1 starts in parallel with the recording. Stages 2, 3, and 4 run after the user clicks stop. The labeling gate (Stage 2) is the one that decides whether the labels read as descriptions of clicks or as descriptions of intent.

Why the labeling gate exists

The labeling gate at check_labeling_gate on line 315 of recording_processor.rs enforces three conditions before step N is allowed to receive its second-pass label. The step itself must have a Stage 1 analysis. The five steps before it must each have an analysis. The five steps after it (or as many as exist if N is near the end of the recording) must each have an analysis. Until all eleven slots are filled, the labeler waits.

The reason is the prompt that runs once the gate opens. Read the prompt verbatim from recording_prompts.rs: "For example, if the target step’s summary is just"Clicked button ‘Submit’", but the neighboring steps show the user filling out a registration form, the new label should be "Submitted the ‘New User Registration’ form"."

Without the gate, the label is correct and useless. With the gate, the label is correct and useful. The synthesis stage that runs next consumes those labels as the names of the steps in the hierarchical workflow, and a workflow named "Click Submit" is much harder to debug than one named "Submit New User Registration" when it breaks in production three months later.

What synthesis is allowed to do, and what it isn’t

Stage 3 takes the full timeline of labeled steps and produces a hierarchical workflow definition: workflows containing steps containing substeps, where each substep declares its inputs, outputs, and business_logic. The schema is enforced by WORKFLOW_SYNTHESIS_SCHEMA in the prompts file, not by post-hoc validation, so the model either returns the right shape or the call fails.

The most load-bearing line of the synthesis prompt is the last rule: "Do Not Hallucinate. Base all synthesized information directly on the provided context and event data. Do not invent steps, inputs, or outputs that are not supported by the evidence." This is a real constraint, not a marketing line. If the user only typed a vendor ID once in the recording, synthesis cannot fabricate a "retry on failure" substep that was never observed; the assistant has no evidence for it. The output is a faithful structured rendering of what actually happened, and nothing else.

That sounds like a limitation, and it is. But it is the same limitation a screen reader has: it can only narrate what is actually on screen. When the workflow needs branching, retry logic, or error handling that the recording did not include, the recording-and-replay assistant is not the right place to add it. The output file (TypeScript via the @mediar-ai/workflow SDK) is, because it is a real file you can edit.

Three perception mechanisms, one outcome that matters

The choice of perception mechanism is the single decision that determines whether your workflow assistant survives a software update. There are three families in the wild today.

The trade-off is honest: the accessibility tree only exists if the OS-level API is wired into the application. On Windows that is most desktop apps. In a browser tab the tree is exposed through the same DOM-based accessibility hooks. On a mainframe terminal emulator, on a Citrix-published session, on a VMWare Horizon view of a remote desktop, the assistant has to fall back to other surfaces. That is where the "AI works on anything" pitch starts to need an asterisk, and the asterisk is worth asking about before signing.

Five questions to ask before buying any workflow assistant

1. Does it read the OS-level accessibility tree, or does it match pixels and selectors. Pixel matching breaks when the icon moves.
2. When it labels a step, does it look at the steps before and after, or only the one in front of it. Out-of-context labels are how recordings turn into garbage.
3. What does it do when the recording is finished. Does it hand you a file you can read, version, and edit, or does it stash the workflow in a black box on the vendor's server.
4. Can the assistant work on a Windows desktop app that has no API. SAP GUI, Jack Henry, Epic, mainframe terminal emulators. The honest answer is yes or no, not, {"it depends"}.
5. What is the unit of billing. Per seat, per minute of execution, per workflow, per agent-hour. The wrong unit makes a cheap assistant expensive at scale.

A vendor that cannot answer these in plain language is selling a chat veneer over an API catalog, not a workflow assistant. The Mediar answers, in order: yes (accessibility tree, no pixels), yes (the [N-5, N+5] gate), yes (a TypeScript file using @mediar-ai/workflow), yes (SAP GUI, Jack Henry, Epic, Cerner, Oracle EBS, mainframe terminals over Win32 emulators), and per-minute of runtime at $0.75/min with no per-seat cost. The source files for everything above are in the open desktop agent at github.com/mediar-ai/terminator.

Frequently asked questions