OCR + Document Understanding Pipeline

Gemini 2.0 Flash is the cheapest vision classifier at $0.075/$0.30 per MTok. Good enough to distinguish invoice/receipt/W-2/ID with 95%+ accuracy. Fine-tune a DistilBERT only if you have 10k+ classifier examples per type.

Alternatives: Claude Haiku 4 vision, GPT-4o-mini vision, Fine-tuned DistilBERT on OCR output

Azure Form Recognizer has the best prebuilt invoice/receipt/ID models and strong multilingual support. Textract is better for pure table extraction. Document AI wins on non-Latin scripts. Tesseract is free but misses tables and struggles with 20%+ of real-world documents. Mistral OCR is promising and cheap.

Alternatives: AWS Textract, Google Document AI, Mistral OCR, Tesseract 5, PaddleOCR

Gemini 2.5 Pro has the best handwriting recognition in 2026 across English and non-Latin scripts. Claude Sonnet 4 is close and better at structured JSON output. GPT-4o is competitive but weaker on non-Latin handwriting.

Alternatives: Claude Sonnet 4 vision, GPT-4o vision

Sonnet 4 produces reliably typed JSON against your schema. Azure prebuilt models (invoice, receipt, W-2) are faster and cheaper when they cover your document type out-of-the-box. Use Sonnet 4 as the fallback and for custom document types.

Alternatives: GPT-4o with Structured Outputs, Gemini 2.5 Pro, Azure prebuilt models

Pydantic catches type errors (dates, currencies, amounts) out of the box. Add custom rules for cross-field checks (sum of line items equals subtotal, tax rate is reasonable for the country, invoice date is not in the future). Validation catches 30-50% of bad extractions before they hit downstream systems.

Alternatives: Great Expectations, OPA rules, JSON Schema

Reviewers need the image side-by-side with structured fields, with click-to-highlight on the source bounding box. Rossum and HyperScience are excellent turnkey solutions for invoices but lock-in and expensive. Build custom if you have 5+ document types or non-standard workflows.

Alternatives: Rossum, HyperScience, Unstructured.io UI

Traditional OCR vs vision LLM end-to-end

Running every document through Gemini 2.5 Pro vision simplifies the pipeline and handles handwriting natively, but costs $0.10-$0.30 per document vs $0.02-$0.05 for Azure Form Recognizer. At low volume (<100k docs/month) or edge cases, vision LLM is fine. At scale, a hybrid (OCR engine for 80%, vision LLM for 20%) is 3-5x cheaper with equal accuracy.

Prebuilt model vs custom extractor

Azure/AWS/Google all ship prebuilt invoice, receipt, and ID models that are 90-95% accurate out of the box. They are cheap and fast but cannot be customized. For custom document types (industry-specific forms, proprietary workflows) or last-mile accuracy gains, a vision LLM with a schema prompt is the right choice.

Confidence threshold - auto-accept vs always-review

Auto-accepting extractions above 95% confidence cuts human review costs 3-5x, but 1 in 300 errors slip through. For AP (accounts payable) that is a $5k wire transfer to the wrong vendor. For medical records, it is a compliance incident. Calibrate the auto-accept threshold per document type based on the downstream cost of an error.

Handwriting misread as similar-looking characters (0 vs O, 1 vs l vs I)

Mitigation: Use character-level confidence scores from the OCR engine. For critical fields (account numbers, amounts, dates), require confidence above 0.9 or route to human review. Run a second pass with Gemini 2.5 Pro vision and compare; disagreements trigger review.

Rotated, skewed, or upside-down photos fail OCR completely

Mitigation: Run a deskew and rotation-detection pass (OpenCV or a small ML model) before OCR. Reject documents below a minimum DPI (150+) with a user-facing error. Mobile uploads should preview the cropped/deskewed version so users can re-scan before submitting.

Amounts extracted with wrong thousand/decimal separator (European 1.500,00 vs US 1,500.00)

Mitigation: Detect the document's locale first (country from address, currency symbol, tax ID format). Apply locale-appropriate number parsing. Validate: line items sum to the stated subtotal - if they don't, the separator was probably misparsed.

PII/PHI leaked into LLM logs and third-party provider

Mitigation: For HIPAA/PII-sensitive documents: use zero-data-retention endpoints (Anthropic ZDR, Vertex AI), sign BAAs, and redact obvious PII (SSNs, DOBs, account numbers) from prompt context when possible. For highest sensitivity (health records, legal docs), self-host Llama 3 or Mistral Large.

Compliance failure - regulator asks for original and extracted side-by-side

Mitigation: Store the original image/PDF, the OCR output with bounding boxes, the extracted fields, model versions, prompt versions, validator results, and any human edits. Keep for the longer of 7 years (SOX), 10 years (HIPAA), or the jurisdiction's tax retention rule. S3 with object lock or Azure Blob WORM is the default.