Repository navigation
Table rows lose their header binding when rendered, making tabular content near-unretrievable #14971
Copy link
Copy link
Open
Labels
area: kb-retrievalWave 5 · cluster J — Knowledge-base retrieval qualityWave 5 · cluster J — Knowledge-base retrieval qualitybackendenhancementNew feature or requestNew feature or requestknowledge-basepriority: mediumrag
Milestone
Description
Problem
Where table content does survive into the knowledge base, each row is rendered without
any binding to its header row. Once chunked and embedded, a row is a sequence of values
whose meaning lives in a different part of the text, or nowhere.
Evidence
autobot-backend/utils/document_parser.py,_parse_docxrenders rows as
" | ".join(row)under a--- Tables ---marker. Row 0 (typically the header) is
rendered identically to every data row, with nothing associating them.autobot-backend/media/document/extraction.py,_normalize_tableproduces a bare
List[List[str]]— a grid with no header concept.- No header-propagation logic exists anywhere in
media/document/.
A row embedded as a bare value sequence is close to unretrievable: the column names
that made it meaningful are not in the vector.
Proposed approach
Heuristic reassembly at render time, no model:
- Treat row 0 as a header row when it is present and non-empty, and emit each data row
as header-to-value pairs so each row carries its own column semantics. - Record which strategy was applied (header-paired vs raw grid) in chunk metadata.
Risks / constraints
- The heuristic is wrong for tables with no header row and for multi-row headers, and
it fails silently — a mislabelled row produces confident nonsense rather than an
error. Recording the applied strategy in metadata is therefore not optional; it is
what makes a bad parse diagnosable, and it is the same discipline as
tables_attempted(fix(document): _extract_pdf_tables stub returns [] while the result claims confidence 0.95 #13895) andOcrResult.attempted(fix(document): media pipeline reports confidence 0.95 success on a zero-text extraction #13884). - Header-paired rendering is more verbose than a pipe join; the size bound from the
parent table-ingest work (PDF tables are extracted on every PDF and dropped by every ingest path #14970) applies here too. - Must not regress fix(knowledge): KB upload loses page provenance — facts cannot cite a source page #13894 — the pairing is meaning, not structure, which is why it may
enter the embedded string where a page marker may not.
Acceptance criteria
- A data row's rendered text contains its column names
- A table with no detectable header falls back to the raw grid rendering
- The applied strategy is recorded in chunk metadata in both cases
- A test retrieves a specific cell value by querying its column name plus a row key
Part of #14967
Blocked by: #14970 (needs tables reaching an ingest path before their rendering matters)
Blocks: Nothing.
Implementation order for the whole umbrella is in #14967.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
area: kb-retrievalWave 5 · cluster J — Knowledge-base retrieval qualityWave 5 · cluster J — Knowledge-base retrieval qualitybackendenhancementNew feature or requestNew feature or requestknowledge-basepriority: mediumrag
Activity