Repository navigation
knowledge: browser empty + 'unvectorized on every reload' — facts (Redis) wiped 43→0 while 22 ChromaDB vectors remain; facts not durably persisted; vectorized status read from Redis flag not the vector store #12733
Description
Activity
- added a commit that references this issue
on Jul 28, 2026 Possible contributor found while tracing #12969, worth checking before diagnosing this independently.
services.knowledge.doc_indexerreports on every boot of the live install:2026-07-29 11:50:37 - WARNING - No documentation files discovered in /opt/autobot/autobot-backendIt sets
self._root_dir = PATH.PROJECT_ROOT(doc_indexer.py:507), which resolves to the backend directory on a deployed host — while the documentation tree is a sibling at/opt/autobot/docs. In a checkoutPROJECT_ROOTis the repo root anddocs/sits under it, so indexing works there and silently indexes nothing in deployment.If any part of "knowledge browser empty" depends on indexed documentation, this would produce exactly that with no error surfaced — the indexer logs a warning and continues.
Root cause and the other three affected consumers are on #12969; it is a
PROJECT_ROOTsemantics problem rather than anything specific to the knowledge base.The general rule behind this is now tracked as #15663 (sub-issue of #13916): no datum in AutoBot declares its system of record, so the Redis-homed facts here are one instance of a repo-wide gap rather than a KB-local bug. This issue stays the concrete fix — #15663 owns the rule and the guard test that stops the next one.
Two of the five recommendations here are now merged, via #15663 / PR #15668 (
324fde1a). Leaving this open — the other three are untouched.Done — recommendation 1 (durable system of record). Facts now live in a
knowledge_factsPostgres table (autobot-backend/models/knowledge_fact.py, migration20260825_085). The durable row is written before the Redis projection (knowledge/facts.py:771-772), and the write is allowed to fail the call: a fact reported stored has been recorded._read_fact_for_writesettles existence from the row when the hash is gone — the exact state that made every fact here read as "not found" while the knowledge was still recoverable — andrebuild_fact_projections()reconstructs everyfact:*key, dedup index, session link and ownership set from the rows.Facts that predate the change are adopted by
adopt_legacy_facts(), wired atknowledge_factory.py:101and:209. That matters for this issue specifically: the migration creates the table empty (the migration gate has no Redis connection), so without the adoption the 22-orphaned-vector deployment would still have had Redis-only facts. It runs on KB construction, so the first backend restart after deploying is when existing facts actually get their rows.Done — recommendation 3 (derive vectorized status).
knowledge/vector_membership.py::vectorized_idsasks ChromaDB directly and returnsNone, not an empty set, when the store is unreachable — so "don't know" cannot render as "nothing is vectorized", which is what produced the "unvectorized on every reload" symptom. Both readers go through it now: the status endpoint (api/knowledge_vectorization.py:47) and — the one that actually kept the damage alive — the reconciler (background_vectorization.py:25, deciding at:119). It previously skipped any fact stampedvectorization_status=completed, so a vector lost underneath a surviving stamp was never rebuilt, in the one component whose job is repairing that drift.repo_tests/store_authority_test.py::test_vectorized_status_is_derived_not_storedfails if either deciding function reads the flag again.Still open here:
- Reconcile the two stores / expose a repair.
rebuild_fact_projections()covers Postgres → Redis andadopt_legacy_facts()covers Redis → Postgres, but neither is reachable from an endpoint, and the 22 orphaned ChromaDB vectors with no fact behind them are not addressed by either. - Find the wipe. Why the facts cleared across the backend restarts is still unknown. The durable store means a repeat is survivable, not that it cannot happen — and if something is issuing a bulk delete, it will now delete the rows too.
- celery: autodiscover_tasks registers nothing (no tasks/tasks.py) — 4 beat-scheduled jobs silently never run, incl. credential reconcile, snapshot cleanup, LLC sprint auto-close #12318 (KB lifecycle tasks unregistered) is unchanged.
The general rule this instance motivated is #15663, now closed: every persisted concept declares its system of record, and
repo_tests/store_authority_test.pyfails when a new one appears without.- Reconcile the two stores / expose a repair.
Summary
The Knowledge Browser (
/knowledge/browser) renders empty, and categories show as unvectorized on every reload. Root cause: the knowledge base's two stores are out of sync and its user data is not durably persisted. Facts (stored in Redis) have disappeared —facts/by_categorywent from 43 facts / ~3 MB earlier this session to 0 now — while ChromaDB still holds the vectors. The KB is internally inconsistent, and the browser (which reads facts) shows nothing.Evidence
So: 0 facts, but 22 documents / 22 vectors in ChromaDB. Categories still exist, so the browser draws the tree structure with no items underneath → it looks empty.
Why the facts vanished
knowledge/facts.py::_store_fact_in_redisuseshsetwith noexpire/setex, so they don't age out.uptime_in_days:4), with RDB persistence on (save 3600 1 …,dump.rdbcurrent) andmaxmemory-policy noeviction— so they were not lost to a Redis restart or eviction.That leaves: the facts were wiped by something on the app side — most likely a backend re-init (there were 10 backend restarts today) that cleared/re-seeded the facts store without restoring, or a delete operation. Either way, user knowledge data was lost, which for a "your data, your AI" product is the most serious failure class.
Durability concern (design)
Knowledge facts are stored in Redis (
redis_db: 1), while their vectors are in ChromaDB (disk). Redis with RDB snapshots is not a durable system of record for user data — a snapshot gap, a re-init, or a flush loses facts, exactly as observed, while the ChromaDB vectors survive (hence the 0-facts/22-vectors split). The system of record for user knowledge should be durable (Postgres/disk), with Redis as a cache — not the sole home of the data."Unvectorized on every reload"
Vectorization status is tracked as fact metadata in Redis (
vectorized_aton the fact hash,knowledge/facts.py:732), not derived from the actual vector store. So:Fix: derive vectorized state from the vector store (is there a vector for this id in ChromaDB?), or reconcile the two stores on read, rather than trusting a Redis-side flag that can drift.
Related — KB cleanup tasks not registered (#12318 still live)
The backend now logs (good — the startup assertion recommended in #12318 landed):
So KB cleanup/expiry tasks are scheduled but never run (they didn't cause this loss, but it means orphan/expiry maintenance is dead — #12318). Reinforces that the KB's store lifecycle is unmanaged.
Recommendation
Impact
Critical — the core Knowledge feature shows empty, user-entered facts were lost, and the vectorized indicator is unreliable. This is the flagship "your data" surface; silent loss of it is the worst outcome for the product's positioning.
Environment
install.shdeploy at/opt/autobot; Redis (facts, db1) + ChromaDB (vectors,autobot_memory). 10 backend restarts today. Related: #12312 (embedding drops — same dual-store drift), #12364/#12365 (dual-mechanism pattern), #12318 (celery tasks unregistered).