Intelligent Document Processing for complex PDFs: contracts, financial reports, medical forms, scanned forms with tables. Layout analysis → OCR → Claude Vision fallback → Pydantic-validated extraction → confidence-gated routing.
Industrial use case: Hyperscience, Rossum, Klarity — IDP for legal-tech, fintech, healthcare, insurance.
Extracts structured fields from messy real-world PDFs. Routes simple text PDFs through unstructured/docling (fast, cheap), scanned ones through OCR (Tesseract/PaddleOCR), and the hard ones through Claude Vision. Validates extractions with Pydantic schemas + cross-field rules. Auto-approves high-confidence; routes the rest to a human reviewer.
PDF input
│
▼
[Document Classifier] Claude Vision → type: contract | invoice | form | report
│
▼
[Layout Analyzer]
├─ text-based → unstructured / docling
├─ scanned → OCR (Tesseract or PaddleOCR)
└─ hard → Claude Vision direct
│
▼
[Table Extractor] Camelot or Table Transformer (TATR)
│
▼
[Field Extractor] Claude + Pydantic schema per document type
│
▼
[Validator]
├─ regex (NIT, dates, amounts)
├─ cross-field (subtotal + tax = total)
└─ LLM-as-judge for coherence
│
▼
[Confidence Scorer] per-field + global
├─ > 0.9 → auto-approve
├─ 0.7-0.9 → human review queue
└─ < 0.7 → re-process with more expensive VLM
- Download FUNSD (199 scanned forms) + DocVQA (12K docs) + PubLayNet (360K pages)
- Document type classifier (Claude Vision)
- Three-path layout analyzer (unstructured | OCR | Vision)
- Table extractor (Camelot for text PDFs, TATR for image-based)
- Pydantic schemas per document type (invoice, contract, form, report)
- Regex + cross-field validators
- Confidence scorer + routing
- Eval: field-level F1 on FUNSD, doc-level accuracy on DocVQA, table F1 on FinTabNet
- Next.js demo with PDF viewer side-by-side with JSON output + confidence per field
- Gallery of 10 pre-processed documents accessible from the demo
| Layer | Technology |
|---|---|
| Parsing | unstructured 0.16+ or docling (IBM, 2024) |
| OCR | Tesseract 5 or PaddleOCR |
| VLM | Claude Sonnet 4.5 with vision input |
| Tables | Camelot or Table Transformer |
| Layout | LayoutParser or detectron2 |
| Schemas | Pydantic v2 |
| Reasoning | Claude Sonnet 4.5 |
| Storage | PostgreSQL |
| Frontend | Next.js with react-pdf viewer |
| Observability | LangSmith |
- Pipeline works on all 3 input types: clean text, scanned, complex layout
- Field-level F1 reported on FUNSD test set
- Document-level accuracy reported on DocVQA
- Table extraction evaluated on FinTabNet or equivalent
- Demo: upload a PDF, see extraction with confidence scores in real time
- Gallery of 10 pre-processed sample documents linked from demo
- Comparative table: unstructured-only vs unstructured + Vision fallback vs Vision direct
Plus the 12 universal DoD blocks.
MIT.