Document processing that sends a human one question, not one document
Two ideas · Confidence · Pakistani formats · The metric · Usage
flowchart LR
D["document"] --> C["classify"]
C --> E["extract fields"]
E --> V["validate the arithmetic<br/>do the totals add up?"]
V --> Q{"which fields<br/>are uncertain?"}
Q -->|"none"| A["auto-process"]
Q -->|"some"| F["route ONLY those fields<br/>to a human"]
style A fill:#16a34a,color:#fff
style F fill:#f59e0b,color:#fff
The difference that decides whether IDP is worth deploying: a human reviews one field, not one document. Validating the arithmetic is what makes per-field confidence mean something.
line items: 12,000 + 8,500 + 3,200 = 23,700
stated total: 27,300
Every field here might have been read at 99% confidence. The document is still wrong — a line was dropped, or the invoice does not add up. Arithmetic consistency catches errors no per-character confidence can see, because each individual read was fine.
That is the cheapest quality signal in document processing and the one most systems skip. Checks here:
- line items sum to the subtotal
subtotal + tax − discount = total- due date does not precede invoice date (a misread year, every time)
- implied tax rate is a statutory rate — 1.7% instead of 17% is a decimal point in the wrong place, the most common numeric OCR error and the least likely to look wrong
A forty-field invoice with one uncertain field is not a failed extraction. Rejecting the whole document sends a human forty fields to re-key when they needed to check one — and that is the difference between a system that saves money and one that quietly costs more than the manual process it replaced.
So the queue carries the specific question:
"total = '99,999.00'. subtotal 23700.00 + tax 4029.00 = 27729.00,
but total states 99999.00"
not "please review this invoice".
Critical fields escalate; non-critical ones do not. A misread address is a nuisance. A misread total is money. Treating them identically is how a review queue becomes a backlog nobody works.
And validation errors escalate regardless of confidence — an invoice whose line items don't sum to its total is wrong even if every character was read perfectly. Confidence has nothing to say about it.
An OCR engine's per-character probability tells you how clearly the ink was printed. It
tells you almost nothing about whether the right value landed in the right field — a
model that reads 12,000 perfectly from the wrong column is 99% confident and completely
wrong.
So confidence is built from things that can be checked:
| Signal | Why it is evidence |
|---|---|
| Agreement | several independent extractors per field; when they disagree the field is uncertain regardless of what any one reported |
| Format | a value satisfying its field's format is more likely to be the right value |
| Position | a number beside Total: beats the same number found loose on the page |
None of these is a probability, and none is presented as one. They combine into a score used for routing. It is capped below 1.0 on purpose — a score of 1.0 invites a downstream system to treat the value as certain, and nothing here can establish that.
CNIC 35202-1234567-1 · NTN 1234567 · STRN 03-02-9999-123-45 ·
IBAN PK36SCBL0000001123456702 · +92 phone numbers ·
amounts as Rs. 11,800.00, PKR 11,800, Rs 1,18,000 (lakh grouping), with the tax
amount taken from GST 18%: Rs. 1,800.00 rather than the rate
Dates parse day-first. Pakistan writes DD/MM/YYYY, and defaulting to month-first
reads 03/04/2026 as 3 April in one system and 4 March in another — silently wrong dates
on contracts, very hard to trace back.
total matched inside "Subtotal". The invoice's total was silently extracted as
its subtotal — a wrong number of the right shape, in the right place, from a real line of
the document. Nothing downstream can detect that. Only a word boundary prevents it.
"Rs. 99.50" failed to parse. Stripping every non-digit character kept the full stop
in Rs., producing ..99.50. Amounts are written that way on most invoices here, so it
is not an edge case. Numbers are now matched, not stripped down to.
document = process(invoice_text, line_items=extracted_rows)
document.auto_approved # True — straight through, no human
document.values # {"invoice_number": "INV-2026-0148", "total": "27,729.00", ...}
document.review_queue # [] or one ReviewItem per uncertain field
document.summary()["questions"]Classification declines two ways, and the second matters more: a document matching nothing is unknown; a document matching two types almost equally is ambiguous. An invoice attached to a contract is a real document and a real problem, and a forced choice there looks exactly like a confident correct answer.
QueueMetrics().summary()
# {"straight_through_rate": 0.82, "items_per_document": 0.31,
# "worst_fields": {"total": 41, "vendor_ntn": 12, ...}}worst_fields is the useful one. Usually a single field accounts for most of the queue,
and fixing that one extractor is the entire win.
python demo.py
Every field in the broken invoice is present, legible and extracted with high confidence. Field-level OCR confidence would pass all eleven. What fails is that 12,000 + 2,040 is not 27,300 — a cross-field check, not a vision one.
That is the whole argument for arithmetic validation: the errors that matter most in finance documents are the ones where the characters were read perfectly.
82 tests with pip install -e ".[api,dev]"; with the base pip install -e ".[dev]"
it is 75 passed + 1 skipped (the API smoke-test module needs the api extra). No OCR
engine, no documents.
| Covered | |
|---|---|
| Parsing | scanned money formats, Rs. prefix, day-first dates, impossible dates |
| Formats | CNIC, NTN, STRN, IBAN, phone; critical vs non-critical severity |
| Cross-field | line-item sum, total arithmetic, date order, implausible tax rate |
| Classification | recognised, unknown, ambiguous, runner-up reported |
| Amounts | Rs. / Rs / PKR / ₨ prefixes, lakh grouping (1,18,000.00), tax amount not the 18% rate, a real-looking Rs. invoice approved when it adds up and flagged when it does not |
| Extraction | word boundaries, longest label wins, agreement raises confidence, never reaches 1.0 |
| Routing | straight-through, per-field not per-document, specific questions, validation overrides confidence, unclassifiable escalates whole, policy strictness |
| Robustness | None/non-string text and malformed line_items are routed for review, never a raw crash |
| Metrics | straight-through rate, worst field, empty queue |
- No OCR. This takes text. Tesseract or a cloud OCR produces it, and the quality of what arrives here bounds everything downstream.
- No table extraction. Line items are supplied separately, because table structure is a different problem and pretending otherwise produces a system bad at both.
- Extraction is label-and-pattern based. It handles the documents it has rules for and
degrades on unusual layouts; a layout model fits behind the same
Extractorinterface. - Confidence is a routing score, not a calibrated probability, and is deliberately not presented as one.
- Three schemas ship (invoice, contract, CNIC). Adding one is a
DocumentSchema. - Windows console: extracted values can contain non-Latin script (an Urdu vendor
name, for example — squarely inside this repo's own Pakistani-documents pitch).
Windows' default console code page (cp1252/cp437) cannot encode that, and a plain
print(doc.values[...])will raiseUnicodeEncodeError.demo.pyreconfigures its own stdout to UTF-8 defensively; in your own script, either do the same (sys.stdout.reconfigure(encoding="utf-8")) or runchcp 65001/ setPYTHONIOENCODING=utf-8first.
intelligent document processing · IDP · document AI · OCR · information extraction · invoice processing · field extraction · confidence calibration · human in the loop · straight-through processing · FastAPI · Pydantic · Pakistani document formats · CNIC · document classification
MIT
git clone https://github.com/hammasbuilds/doc-intelligence-api
cd doc-intelligence-api
pip install -e ".[dev]" # the library has zero dependencies; dev adds pytest, ruff, httpx2
pytest -q # 75 passed, 1 skipped; 82 passed after pip install -e ".[api]"
python demo.py # the Output section below, reproducedfrom docintel import process, ReviewPolicy, QueueMetrics
document = process(invoice_text, line_items=extracted_table_rows)
document.auto_approved # True -> straight through, no human
document.values # {"invoice_number": "INV-2026-0148", ...}
document.summary()["questions"] # one precise question per uncertain field
metrics = QueueMetrics()
metrics.add(document)
metrics.summary()["worst_fields"] # which extractor to fix firstThis takes text, not images — Tesseract or a cloud OCR produces it, and line items are supplied separately because table extraction is a different problem.
pyproject.toml declared a FastAPI api extra from the start; this is the actual demo
that extra was for. Paste or pick a sample document, see it classified, extracted,
validated, and either auto-approved or routed to a human with a specific question per
field — plus a running dashboard (straight-through rate, worst fields by queue volume)
that accumulates across everything processed in the session.
pip install -e ".[api]"
uvicorn docintel.api:app --reload --app-dir src
# open http://127.0.0.1:8000Local only, in-memory metrics — this is a demo of the library above, not a deployed
service. The paste box is capped at Starlette's default of 1MB per form field; a
document larger than that (a long multi-page contract, say) gets a friendly in-page
message rather than a raw error — call docintel.process() directly for anything
bigger, it has no such limit.