Repository navigation
bug(kb): add_document never chunks — a whole transcript or PDF becomes one embedding, against its own docstring #17830
Description
What's broken
KnowledgeBase.add_document() embeds the entire document as a single vector. The class docstring claims otherwise, and eight production call sites — including whole meeting transcripts and the manual-upload endpoint — rely on it.
Evidence
autobot-backend/knowledge/documents.py:30— "Add documents with chunking";:37— "Automatic chunking and vectorization".:42-75add_document()adds onlyasyncio.wait_fortimeout protection, then calls_add_document_internal(content, metadata, doc_id).:77-92_add_document_internal()sets two provenance defaults and returnsawait self.store_fact(content, metadata, doc_id). No splitting step anywhere in the file.knowledge/facts.py:690-744_vectorize_fact_in_chromadbthen embeds the whole string:embedding = await _generate_embedding_with_npu_fallback(content, ...)(:718) andDocument(text=content, ...)(:739).
Production callers (grep -rn '\.add_document(' autobot-backend/, excluding tests):
api/knowledge_maintenance.py:558, api/transcripts.py:229, api/knowledge_mcp.py:512, api/workflow.py:124, transcriber/knowledge/kb_push.py:60, orchestration/workflow_documentation.py:116 and :274, and knowledge/documents.py:181 (add_document_from_file, i.e. every PDF/TXT/MD upload).
Failure scenario
A 40-page PDF or an hour-long transcript is uploaded. One embedding is computed for the whole text, so it represents the average of everything in it and matches no specific query well. Retrieval for a question answered on page 31 either misses the document or returns the entire text as one oversized "chunk", displacing better-matched facts under the context budget. Nothing errors; recall silently degrades.
This is worth contrasting with the paths that do chunk properly, which is why the gap is easy to miss: autobot_shared/doc_chunking.py:98-189 (header-then-paragraph with a token budget, used by the system-docs indexer) and knowledge/pipeline/extractors/semantic_chunker.py:74-121 (paragraph→sentence with token overlap, used by the ECL pipeline) are both structure-aware. The general "add a document" path reaches neither.
Acceptance criteria
-
add_documentchunks before storing, reusingautobot_shared/doc_chunking.pyrather than adding a fourth chunker (utils/text_chunking.pyalready exists as a character-based third). - Chunks from one document share a document id / provenance so they can be grouped and cited back to their source.
- A test asserts that a document above the chunk budget produces more than one stored fact.
- If single-vector storage is intentional for some callers, the docstrings at
:30and:37are corrected and the intent is recorded — a docstring claiming a control that does not exist is the defect that hid this.
Found during an external-source comparison recorded in docs/research/private-tenant-chat-reference-app.md (finding F6).
Activity
docs(research): private-tenant chat reference app (#17826) (#17835)
guards(api): bind the pass-through floor to reach, add the two missin…
fix(kb): route upload, facts and url through add_document with per-ch…
fix(kb): page_spans computed from the finally stored text; contrast t…