Repository navigation
fix(knowledge): migrate Chroma collection on embedding-dimension mismatch (#10420) - #10439
Merged
Merged
Conversation
…atch instead of dropping all chunks (#10420) A ChromaDB collection's vector dimension is fixed at creation. When the active embedding model's dimension changes, get_or_create_collection returns the OLD collection and every upsert is rejected — flooding backend-error.log with one "Dropping chunk ... Collection expecting embedding with dimension of N, got M" per chunk and silently breaking knowledge indexing (observed live: 768 vs 3072). Detect the mismatch at init and either recreate the collection (opt-in AUTOBOT_KNOWLEDGE_AUTO_REINDEX_ON_DIM_MISMATCH=true, destructive → re-index) or raise once with an actionable message — instead of the per-chunk flood. Robust dimension resolution (the original guard only worked for _KNOWN_DIMS models, so the live 3072-dim model silently bypassed it): - active dim: org config > _KNOWN_DIMS > live probe of the embedding model - existing dim: provenance metadata > peek an actual stored vector (legacy collections created before provenance metadata) 30 unit tests (probe, legacy-peek, config-dim, the 768→3072 case, flag on/off). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
✅ SSOT Configuration Compliance: Passing🎉 No hardcoded values detected that have SSOT config equivalents! |
10 tasks
Applied Black, isort, and autoflake to match code-quality checks. Triggered by workflow auto-fix-formatting.yml. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thinking Path
While validating "log errors solved in codebase" on the live box,
backend-error.logwas flooded byservices.knowledge.doc_indexer—Dropping chunk … Collection expecting embedding with dimension of 768, got 3072(one per chunk). A ChromaDB collection's dimension is fixed at creation; when the configured embedding model changes dimension,get_or_create_collectionreturns the stale collection and every upsert is rejected → knowledge indexing silently broken + log flood.What Changed
services/knowledge/doc_indexer.py:AUTOBOT_KNOWLEDGE_AUTO_REINDEX_ON_DIM_MISMATCH=true→ delete + re-index, since old vectors are unusable) or raise once with an actionable message — never the per-chunk flood._KNOWN_DIMS, so the live 3072-dim model bypassed it entirely:_KNOWN_DIMS→ live probe of the embedding modelVerification
pytest services/knowledge/test_doc_indexer_dim_mismatch.py→ 30 passed (probe, legacy-peek, config-dim, the exact 768→3072 case, flag on/off, idempotency).flake8 --max-line-length=120clean.Model Used
Claude Opus 4.8
Closes #10420.