Designing document automation that survives messy inputs

01

Treat the input envelope as part of the problem

Inventory the real channels and formats before selecting an extraction approach. Email attachments, uploaded scans, office files, photographs, and generated PDFs behave differently. Some PDFs contain text; others are only images. Password protection, corruption, and oversized files need an explicit path.

Preserve the original file and assign a stable identifier before conversion. Derived text, page images, extracted fields, and reviewer changes should point back to that identifier. This creates a chain from a structured value to the document that produced it.

02

Separate recognition from interpretation

Optical character recognition answers what marks appear on a page. Interpretation answers which marks represent an invoice number, obligation, address, or date. Combining both into one opaque step makes failures difficult to diagnose.

Keep intermediate text and layout information where practical. When a field is wrong, the team can determine whether recognition failed, the field mapping was ambiguous, or the source itself was inconsistent. That distinction directs the next improvement.

03

Define a schema with evidence

A useful output schema includes more than field names. Specify type, required status, accepted formats, normalization rules, and what should happen when a value is absent. Do not silently convert uncertainty into an empty string or a guessed value.

For important fields, capture a source reference such as page number and a short supporting excerpt or region. Reviewers should be able to jump from a proposed value to the evidence. Provenance reduces review time and makes disputes easier to resolve.

• Raw value exactly as represented in the source
• Normalized value used by downstream systems
• Page or section reference
• Confidence or review status
• Validation messages and reviewer correction

04

Use deterministic validation after extraction

Language models can interpret variation, but ordinary code should enforce rules that ordinary code can express. Validate date formats, totals, required identifiers, allowed values, arithmetic relationships, and cross-field consistency after extraction.

Validation should not merely reject a record. Return a clear message tied to the affected field and source evidence. A reviewer needs to know whether a value is missing, malformed, inconsistent with another page, or simply outside the expected range.

05

Design the exception queue first

Every document workflow has cases the automation should not finalize. Build the review state and escalation reason before pursuing full straight-through processing. Common reasons include unreadable pages, conflicting values, missing required sections, unexpected document types, and policy-sensitive content.

The queue should display the original, proposed fields, evidence, validation messages, and a compact correction action. Record the final decision and correction without overwriting the initial output. Those corrections become valuable evaluation examples later.

06

Evaluate by field, document, and workflow outcome

A single accuracy number hides important differences. Measure exactness for critical fields, document-level completion, false acceptance, false escalation, reviewer correction time, and the share of items that can proceed safely. Weight fields according to consequence rather than convenience.

Use a sample that includes routine and adversarially messy inputs. Re-run it after OCR, prompt, model, or schema changes. The goal is not a perfect extraction score; it is a controlled workflow where errors are visible, reviewable, and prevented from quietly reaching the next system.