Hierarchical retrieval and table ingest: two capabilities built and discarded, three quality gaps #14967

Problem

A review of AutoBot's document-ingest and hierarchical-retrieval paths found that
two capabilities we already pay for are being discarded before they reach retrieval,
plus three narrower quality gaps in how hierarchical summaries and tabular content are
built and embedded.

The two discard findings are the priority: in both cases the expensive producer is wired
and runs on every document, and no consumer reads the result. This is the "full surface,
no sink" class.

Confirmed findings (evidence in each child)

Children

Sequencing

These are two independent tracks plus one standalone — they do not form a single chain,
and the two entry points can be worked in parallel.

graph LR
  A["#14968<br/>RAPTOR provenance"] --> B["#14969<br/>RAPTOR sink"]
  B --> C["#14972<br/>order-aware clustering"]
  D["#14970<br/>PDF tables reach ingest"] --> E["#14971<br/>table header binding"]
  F["#14973<br/>branching refinement"]

Entry points: #14968 and #14970. Both are unblocked today.

Highest value first: #14969 and #14970 are the two findings where we already pay the
production cost and read nothing back. #14968 is sequenced ahead of #14969 only because it
is a precondition, not because it is independently valuable.

What this review found we already do well — do not "fix" these

• Provenance is deliberately kept out of the embedded string. render_plain
  (fix(knowledge): KB upload loses page provenance — facts cannot cite a source page #13894) returns (text, Tuple[PageSpan, ...]) with no page markers, so structure
  does not compete with the document's own words for similarity. Any table or tree
  work must preserve this property, not regress it.
• Parse failure is instrumented to be loud, not silent. has_usable_text_layer,
  empty_page_numbers, text_page_ratio, avg_chars_per_page, tables_attempted
  (fix(document): _extract_pdf_tables stub returns [] while the result claims confidence 0.95 #13895), OcrResult.attempted (fix(document): media pipeline reports confidence 0.95 success on a zero-text extraction #13884), and a 0.0 confidence for a
  no-text-no-tables document. This is a genuine strength and the reason several
  candidate techniques below were rejected.
• OCR is resource-bounded on four axes: per-page wall-clock, whole-document
  wall-clock, page count, and per-page pixel budget, with explicit MemoryError
  handling — all env-var-backed.
• Reranking is five-factor (reranker, vector, edge, recency, staleness) plus
  provenance_adjustment and an MMR diversity pass (feat: Configurable reranker blend weights (Neural Mesh RAG Phase 1) #2004, Enhancement: MMR (Maximal Marginal Relevance) diversity scoring in reranker #2090).
• Graph ranking uses personalised PageRank — per-query seed set, power iteration,
  convergence tolerance, hop-bounded subgraph expansion.
• Sufficiency-checked retrieval already exists and is wired (AdaptiveRAGRefiner,
  feat(rlm): adaptive RAG query refinement using RLM evaluator #1382). Only its shape is at issue — see child 6.

Deliberately NOT proposed — do not re-raise

• Vision-model layout classification (classifying page regions into header /
  footer / caption / reference / equation / table with a trained model). Rejected:
  it reintroduces exactly the silent-failure class our document layer was built to
  eliminate — a mis-detected region returns plausible text, so a bad parse reads as
  a bad answer rather than an error — and it adds model weights plus an inference
  budget to the ingest hot path. A positional heuristic for repeated first/last
  lines across pages is the cheap 80% and can be raised separately if header/footer
  dilution is ever measured as a real problem.
• Unconditional scanned-page rotation retry (try four orientations, pick by OCR
  confidence). Rejected as a default: it is a 4x multiplier on precisely the resource
  media/document/ocr.py most carefully bounds. Acceptable only if scoped to pages
  that already produced no text and drawn from the existing page and document
  budgets — not worth filing until scanned-rotation failures are actually observed.
• Replacing our graph ranking with a static global PageRank feature. Our
  personalised, per-query-seeded implementation is strictly stronger.
• Adopting an external ingest/retrieval stack wholesale. The operational floor
  (a relational store, a cache, an object store, and a separate search engine, plus
  a host-kernel prerequisite, before any model loads) is a large regression against
  what we reach today on our existing stores, to acquire capabilities we mostly
  already have.

Acceptance criteria

• All six children closed with evidence, or explicitly declined with a reason
  recorded here
• No regression to the marker-free embedding property (fix(knowledge): KB upload loses page provenance — facts cannot cite a source page #13894)
• No regression to the parse-honesty instrumentation (fix(document): media pipeline reports confidence 0.95 success on a zero-text extraction #13884, fix(document): _extract_pdf_tables stub returns [] while the result claims confidence 0.95 #13895)