OCR recovers text; layout extraction also identifies relationships such as table rows and selected boxes. A vision-capable model can interpret page images alongside text. Choose between them by testing the fields your workflow requires on representative documents, including the effort needed to correct each result.
A consultancy extracting commitments from scanned client reports has a different problem from a team making an archive searchable. Both receive PDFs. Only one needs to distinguish a proposed delivery date from an approved one in the adjacent column.
A summary can read well even when the extracted date belongs to the wrong column. Check the destination field against the page before judging the summary.
OCR, layout extraction and multimodal models return different things
OCR converts visible characters into machine-readable text. Microsoft’s Document Intelligence Read model extracts printed and handwritten text from scanned images and PDFs, with outputs including words, locations and word-confidence scores.
Layout extraction adds explicit document structure. The Document Intelligence Layout model combines OCR with models that extract tables, selection marks and structural elements. Treating this as plain OCR hides a useful middle option between a text transcript and direct model interpretation.
Multimodal input gives a model visual evidence to interpret. For PDF inputs on vision-capable models, OpenAI’s file-input documentation says the API supplies both extracted text and page images. The model can therefore receive page information that a flattened transcript leaves out. This does not establish that it will extract every field correctly.
These routes can also be combined. An extraction service can supply text and table structure, while a language model maps the result into the fields your system expects. Retaining that intermediate extraction lets a reviewer distinguish a misread character from a correctly read date assigned to the wrong column. Fixing the responsible step avoids changing the model when the page representation caused the error.
| Route to test | Starting use case | Failure to look for | Evidence to retain |
|---|---|---|---|
| OCR text with validation | Searchable archive or text fields whose labels remain unambiguous | Misread characters, lost reading order, omitted text | Original page and extracted word or line |
| Layout extraction with field mapping | Tables, checkboxes and repeated forms where location identifies meaning | Correct text attached to the wrong row, column or label | Table structure, page location and mapped field |
| Multimodal model with a defined output schema | Visually varied reports requiring interpretation across text and images | Plausible field assignment unsupported by the page | Source page, extracted value and the evidence used to assign it |
| Combined route | Mixed document families with different failure patterns | Disagreement between extraction and interpretation | Both intermediate outputs and the final acceptance decision |
Use these starting points to choose trial candidates. None establishes a vendor’s accuracy on your documents.
A perfect transcript can still put the wrong date in your system
Consider an illustrative client report containing two columns: “proposed completion” and “approved completion”. The approved cell is blank. The proposed cell contains a date.
OCR can recover every character accurately while a later step flattens the table into a sequence of words. If the field-mapping model fills approved_completion with the only date it finds, the transcription succeeded and the business extraction failed.
A layout-aware route retains the column relationship explicitly. A multimodal route supplies the page image for interpretation. Either still has to demonstrate that it leaves the approved date empty when the document does.
Write that requirement before testing. Otherwise a comparison rewards whichever system fills the most fields, including fields that should remain blank.
For a project team, the downstream consequence is concrete: a proposed date enters a delivery tracker as an agreed commitment. Preventing that error is more valuable than producing a smoother paragraph about the report.
Compare identical documents against a field-acceptance worksheet
Build the evaluation from documents the workflow actually receives. Include a clean digital PDF as a control, then scans with the variations that matter to the work: rotated pages, small print, tables split across pages or empty cells. Use authorised samples and the deployment settings your organisation permits.
For every document, a person should label the required values and their page evidence before comparing routes. Keep a separate set of unseen documents for the final check; prompt tuning against the same examples does not establish performance on the next client report.
Use this worksheet for each extracted field:
| Check | Acceptance rule | Record on failure |
|---|---|---|
| Value | Matches the labelled value under agreed normalisation rules | Character error, incorrect date or other wrong value |
| Association | Belongs to the intended label, row and document | Correct text mapped to the wrong business field |
| Absence | Remains empty when the value is absent | An unsupported value filled into a blank field |
| Evidence | Reviewer can locate the supporting page and relevant area | Missing page reference or evidence that supports a different field |
| Readiness | Required fields pass the workflow’s checks before use | Exception route and time needed to resolve it |
Keep consequential identifiers exact unless an explicit rule permits transformation. Preserve both the source string and any normalised value. A date format conversion should be inspectable; an ambiguous date should enter an exception route rather than acquire an assumed meaning.
Report results by document family and critical field. For the report example, count incorrect approved dates separately from errors in descriptive text. Before any write to the delivery tracker, require the approved-date field to match its labelled source or pass human review. An aggregate accuracy figure conceals this failure if easy descriptive fields dominate the test set.
The worksheet is a proposed evaluation protocol. Model Operator has not executed a benchmark of these products.
Confidence scores require a workflow-specific acceptance rule
Microsoft’s confidence guidance distinguishes confidence at different extraction levels and notes that not all fields return a score. It also recommends considering human review for critical workflows.
A high score for a recognised word does not establish that your application assigned it to the right business field. Check transcription and mapping separately. Avoid treating a confidence score from one extraction service as interchangeable with a language model’s written claim that it is confident.
Set an auto-accept rule against labelled results for the particular document family and consequence. Track incorrect auto-accepts separately from cases sent for review. Lowering the review count has little value if more wrong commitments enter the receiving system.
Price the route after counting the correction work
OpenAI documents that supplying extracted text and page images consumes context and can increase token usage. Measure usage on the page lengths and scan quality your team receives before estimating the monthly bill.
Compare extraction charges and model usage alongside retries, reviewer time and unresolved exceptions. For the trial, define an accepted document as one whose required fields pass the worksheet, after any recorded correction. Divide total processing and review cost by those accepted documents; report documents still awaiting resolution separately. This exposes a route that saves on API charges by passing table reconstruction to a person.
The AI agent implementation cost guide explains the wider budget boundary. This document-processing step also does not require an autonomous agent by default: a fixed extraction, validation and review sequence can be sufficient. Use the agents-versus-workflow-automation comparison to decide where adaptive judgement earns its cost.
Once the evidence is collected, select the simplest route that meets the critical-field acceptance rules at a sustainable operating cost. Keep an exception path for documents outside the tested families.
Model Operator is a Dubai-based AI product studio led by Alexander Chrisostomou. Its AI Initiative Consulting engagement covers workflow analysis and readiness decisions; discovery determines the implementation scope. To decide how document extraction should fit a wider internal workflow, start a Model Operator build conversation with one representative document family and the fields the receiving team must trust.