Bug: Chat halluccinates instead of using knowledge base for AutoBot documentation queries #1385

Activity

1. mrveiss commented on Mar 5, 2026

   

   mrveiss

   on Mar 5, 2026

   OwnerAuthor

   More actions

   Fix Applied

   1. System prompt updated

   Added anti-hallucination policy to resources/prompts/chat/system_prompt_simple.md:

   • Every factual claim must come from KB context, command output, or web search
   • When no KB context is available, explicitly say so and search the web
   • Rule Docs improvements #1: "NEVER hallucinate"

   2. Documentation indexed

   Ran index_documentation.py --full on .20:

   • 2913 documents indexed into autobot_docs ChromaDB collection
   • Query "how to install autobot" returns 3 results with scores 0.73-0.75
   • DocumentationSearcher confirmed working: recognizes doc queries and retrieves relevant chunks

   3. Backend restarted

   Restarted to pick up the new system prompt. Waiting for init to verify end-to-end.

   Root Cause Chain

   1. autobot_docs ChromaDB collection was empty (0 documents) → DocumentationSearcher.search() returned nothing
   2. _retrieve_documentation_context() returned empty → no doc context in LLM prompt
   3. autobot_memory KB also had no install docs → conversation_aware_retrieve() returned empty
   4. System prompt said "General questions → Answer helpfully" → LLM fabricated answer from training data
2. mrveiss

   added a commit that references this issue on Mar 5, 2026

   feat(knowledge): consolidate doc indexing with ChromaDB as single sou…

   ...

   19daec5
3. mrveiss commented on Mar 5, 2026

   

   mrveiss

   on Mar 5, 2026

   OwnerAuthor

   More actions

   Resolved — Commit 19daec5

   Root Cause

   Two disconnected doc indexing systems existed:

   1. populate_autobot_docs API → stored whole files as Redis KB facts (autobot_memory)
   2. index_documentation.py CLI tool → stored chunked docs in ChromaDB (autobot_docs)

   The ChromaDB collection was never populated because the CLI tool was never wired into any lifecycle or API. When the chat RAG pipeline queried autobot_docs, it found 0 documents and the LLM hallucinated answers.

   Fix

   Consolidated into a single system with ChromaDB as the single source of truth:

   1. New DocIndexerService (services/knowledge/doc_indexer.py) — extracted from CLI tool, adapted for service use (SSOT config, centralized ChromaDB client, incremental SHA-256 hash cache)
   2. Refactored populate_autobot_docs endpoint — now delegates to DocIndexerService.index_all() instead of Redis KB store_fact()
   3. Auto-index on startup — background task in lifespan.py fires if autobot_docs collection is empty (non-blocking)
   4. Updated documentation watcher — uses DocIndexerService.index_file() for real-time reindexing on file changes
   5. Anti-hallucination guardrail — added "No Hallucination Policy" to system prompt requiring KB/command/web grounding

   Verification

   • Backend healthy on .20, doc_searcher initialized with 2913 documents
   • Search for "how do I install autobot" returns relevant results (scores 0.74-0.75)
   • POST /api/knowledge_base/populate_autobot_docs returns success with correct counts
   • Background indexing completes without blocking startup
4. mrveiss

   closed this as completedon Mar 5, 2026