AI form recognition and compliance: which step is the AI, and what does the auditor read?

The two pipelines

Strip the marketing off the category and there are two pipelines hiding under one phrase.

Pipeline one is source recognition. A PDF arrives. A scan arrives. A photo of a paper claim arrives. A model converts pixels and text into a structured record. This is where Azure AI Document Intelligence, AWS Textract, Google Document AI, Rossum, Docparser, Affinda, and Nanonets sit. The thing being recognized is the source. The output is JSON or a row in a queue. The compliance question on this pipeline is the standard one for any AI inference: confidence thresholds, human review for the long tail, model cards, retraining cadence, blind sampling. Section 164 of HIPAA does not name OCR specifically, but the Office for Civil Rights audits the process the covered entity wraps around it.

Pipeline two is destination recognition. A claims rep needs a value typed into screen four of a Guidewire first notice of loss. A patient access coordinator needs a field set in an Epic registration session. A bank associate needs a record opened in Fiserv DNA. The thing being recognized is the live form in the desktop application. There is no source image here; the form is the Win32 control. The compliance question on this pipeline is different: what reads the form, what writes to it, and what audit artifact does the writer leave behind. A model that hallucinates a field name is a much bigger problem here than in the source pipeline because there is no human reviewer staring at the JSON in between.

What changes when the AI moves from the source to the destination

The toggle below is the same data, recognized two ways. On the left, an OCR pipeline reads a CMS-1500 PDF and emits JSON. On the right, an accessibility-tree pipeline reads the corresponding fields in the live Epic registration screen and types values into them. Same patient, same record, two completely different audit shapes.

The point is not that one pipeline is better than the other. The point is that they have different audit shapes, and the answer to “is AI form recognition compliant?” depends on which one a buyer is actually choosing.

Why HIPAA and SOX care about determinism on the destination side

The regulator is not auditing the model. The regulator is auditing whether the same input always produced the same record. That property is cheap to argue when the writer is a SQL stored procedure, a payroll calculation, or a cron job. It is harder when the writer is a probabilistic model that picks the next click on the fly. Two identical claims can produce two different field values. Two identical KYC packets can produce two different beneficiary records. Two identical journal entries can produce two different account postings. None of those are illegal by themselves; all of them are difficult to defend in a control walkthrough.

A deterministic runtime restores the property. The workflow file is the testable unit, the same way a database migration is the testable unit for a schema change. A reviewer reads it, redlines one step, runs it against a test environment, and signs it. Whatever model wrote that file in the first place is gone by the time the reviewer opens the artifact.

The audit artifact

What an auditor opens on the destination side, in literal form, is a TypeScript file. Each step is a structured record with the field names the live application uses. A reviewer reads it, diffs it, and signs it. The runtime executes that file byte for byte on each run.

The locator resolver lives in apps/desktop/src-tauri/src/focus_state.rs. It walks four strategies in order: the recorded automation id, the window handle plus bounds, the visible text content, and the parent window as a last fallback. Three of those four do not depend on absolute pixel position, so a routine UI tweak (a button moves down a row, a panel reorders, a form gains a tab) usually still resolves through strategies one, two, or three. Only when all four miss does the step fail loudly into the trace. The failure is queueable for re-recording, not silently retried with a guessed element.

The split, in numbers

Counts come from the open-source Mediar repo. Zero LLM call sites in crates/executor (a single ripgrep on github.com/mediar-ai/terminator confirms it). Four locator strategies in apps/desktop/src-tauri/src/focus_state.rs. Four call sites for vertex_ai::call_vertex_ai inside recording_processor.rs, all of which run in the offline recording pass. Eight semantic fields per step in the recorded workflow format. The split between the offline AI and the runtime is mechanical, not aspirational.

What the auditor walkthrough actually looks like on each side

The two checklists below are the two control walkthroughs an internal auditor or external assessor would do for the two pipelines. They are not interchangeable. A team that runs an OCR pipeline on the source and skips the second walkthrough still has a regulated form in their system of record being filled by something. A team that runs a UIA pipeline on the destination and skips the first walkthrough still has a vision model on the source PDF.

Boxes left unchecked deliberately: each item is a real obligation the buyer takes on when they choose an OCR-based source pipeline. They are not failures, they are the price of the probabilistic step.

Boxes pre-checked here for a different reason: every item is something the workflow file already exposes, so the obligation on the buyer is to read and sign the file, not to maintain a separate model governance layer.

Counterargument: when an OCR-shaped answer is the right one

The honest case for an OCR-only pipeline is real and worth naming. If the only thing the buyer needs is to turn a stack of PDFs into a CSV that lands in a data warehouse, the destination pipeline does not exist as a problem. A vision model on the source, a confidence threshold, a review queue for the long tail, and a documented retraining cadence is a complete answer. Azure AI Document Intelligence, AWS Textract, Google Document AI, Rossum, and Nanonets all do this work well. The tooling is mature and the audit shape is well understood.

The mistake is buying that tool and assuming it has also recognized the form on the destination side. It has not, because the surface it integrates against simply does not exist on a Win32 application. The two pipelines want different products. A reasonable stack uses one tool from each side and is honest about the audit shape of both.

Resolution

So back to the literal question. Is AI form recognition compliant? On the source, yes, with the standard OCR governance package: per field confidence, human review queue, model card, blind sampling, retraining cadence. On the destination, the question is the wrong shape: there is no probabilistic step to govern, because the deterministic answer is to read the Windows UI Automation tree directly and ship the workflow file as the artifact. The model is gone before the auditor walks in.

A compliance team that buys one tool and ignores the other layer ends up with either a beautiful PDF extraction and a regulated form being filled by something they cannot describe, or a beautiful workflow file and a backed-up queue of source documents. Neither half on its own is a form-recognition strategy under regulation. The right answer is one tool from each layer, with a clean hand-off, and a clear story for what the auditor reads at each end.

Frequently asked questions