Skip to content

bug(kb): add_document never chunks — a whole transcript or PDF becomes one embedding, against its own docstring #17830

Open

Description

@mrveiss

What's broken

KnowledgeBase.add_document() embeds the entire document as a single vector. The class docstring claims otherwise, and eight production call sites — including whole meeting transcripts and the manual-upload endpoint — rely on it.

Evidence

  • autobot-backend/knowledge/documents.py:30 — "Add documents with chunking"; :37 — "Automatic chunking and vectorization".
  • :42-75 add_document() adds only asyncio.wait_for timeout protection, then calls _add_document_internal(content, metadata, doc_id).
  • :77-92 _add_document_internal() sets two provenance defaults and returns await self.store_fact(content, metadata, doc_id). No splitting step anywhere in the file.
  • knowledge/facts.py:690-744 _vectorize_fact_in_chromadb then embeds the whole string: embedding = await _generate_embedding_with_npu_fallback(content, ...) (:718) and Document(text=content, ...) (:739).

Production callers (grep -rn '\.add_document(' autobot-backend/, excluding tests):
api/knowledge_maintenance.py:558, api/transcripts.py:229, api/knowledge_mcp.py:512, api/workflow.py:124, transcriber/knowledge/kb_push.py:60, orchestration/workflow_documentation.py:116 and :274, and knowledge/documents.py:181 (add_document_from_file, i.e. every PDF/TXT/MD upload).

Failure scenario

A 40-page PDF or an hour-long transcript is uploaded. One embedding is computed for the whole text, so it represents the average of everything in it and matches no specific query well. Retrieval for a question answered on page 31 either misses the document or returns the entire text as one oversized "chunk", displacing better-matched facts under the context budget. Nothing errors; recall silently degrades.

This is worth contrasting with the paths that do chunk properly, which is why the gap is easy to miss: autobot_shared/doc_chunking.py:98-189 (header-then-paragraph with a token budget, used by the system-docs indexer) and knowledge/pipeline/extractors/semantic_chunker.py:74-121 (paragraph→sentence with token overlap, used by the ECL pipeline) are both structure-aware. The general "add a document" path reaches neither.

Acceptance criteria

  • add_document chunks before storing, reusing autobot_shared/doc_chunking.py rather than adding a fourth chunker (utils/text_chunking.py already exists as a character-based third).
  • Chunks from one document share a document id / provenance so they can be grouped and cited back to their source.
  • A test asserts that a document above the chunk budget produces more than one stored fact.
  • If single-vector storage is intentional for some callers, the docstrings at :30 and :37 are corrected and the intent is recorded — a docstring claiming a control that does not exist is the defect that hid this.

Found during an external-source comparison recorded in docs/research/private-tenant-chat-reference-app.md (finding F6).

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions