You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
reindex --search cannot repair DB-only search index divergence #1226
basic-memory reindex --search cannot repair database-only divergence: it selects files to
index purely from file-content checksum changes, so when the observation/relation/ search_index tables disagree with an unchanged markdown file, the incremental pass selects
zero files and reports success while search keeps serving stale rows.
Field report on #1214 (SQLite, concurrent MCP clients): after trimming duplicate observation rows, search_index still held 3886 rows of type='observation' against 2146
observation rows. reindex --search reported Indexing files... 0/0 files and changed
nothing; until reindex --full --search ran, search answered with content no longer in the
store.
Root cause
select_project_index_target_files (src/basic_memory/indexing/project_index_coordinator.py:229-242)
builds its target set from change_report.new_files | modified_files, which change_detector.py derives purely from file checksums (load_indexed_file_checksums).
DB-only divergence leaves every file in unchanged_files, so nothing reaches search_service.index_entity_data — the only code path that deletes and rewrites an
entity's observation search rows. _purge_stale_search_rows only handles rows for deleted
entities, and only runs in the --embeddings path.
Proposed direction
Add a DB↔FTS reconciliation sweep to the search reindex path (extend _purge_stale_search_rows or a sibling): purge search_index rows whose id no longer
exists in the owning table (type='observation' → observation.id, type='relation' → relation.id), and optionally detect entities whose row counts disagree and re-enqueue just
those entities, so --search without --full converges without a full rebuild.
The one-time damage from the observation duplication bug is handled by the #1214 migration's
FTS orphan purge; this issue is about making incremental reindex --search able to repair
such divergence in general.
Acceptance criteria
reindex --search (no --full) removes search_index rows of type='observation'/type='relation' whose backing row no longer exists.
A seeded DB↔FTS divergence on an unchanged file converges via reindex --search alone (regression test).
No full-file re-parse is required for the purge path; --full behavior unchanged.
Type: Bug (confirmed — bug label already applied) Component: Database/sync — reindex --search indexing pipeline (src/basic_memory/indexing/project_index_coordinator.py); not cloud-specific Priority: High — search silently serves stale/incorrect content after DB-only divergence, and the documented incremental repair path (reindex --search) reports success while doing nothing. No data loss, but a correctness/trust issue for search results. Complexity: Medium — the fix is scoped (extend the FTS reconciliation sweep already present for the --embeddings/deleted-entity path to also run for --search, purge orphaned search_index rows by id, and add a regression test for a seeded DB↔FTS divergence on an unchanged file). No architectural changes required, but touches the _purge_stale_search_rows / search_service.index_entity_data interaction so needs careful testing to avoid affecting the unrelated --full path.
Verification: Read select_project_index_target_files in src/basic_memory/indexing/project_index_coordinator.py:229-242 — it does build its target set solely from change_report.new_files | modified_files, which come from file-checksum diffing (change_detector.py), so a DB-only divergence (no file change) indeed produces an empty target set and reindex --search no-ops. Root cause as described in the issue is accurate.
Duplicate check: No existing open or closed issue duplicates this. #1214 is a related companion issue (generation-versioned observation persistence / the field report that surfaced this), not a duplicate — already correctly cross-referenced in the issue body.
Activity
claude commentedon Aug 11, 2026
Claude finished @phernandez's task in 41s —— View job
Triage
Type: Bug (confirmed —
buglabel already applied)Component: Database/sync —
reindex --searchindexing pipeline (src/basic_memory/indexing/project_index_coordinator.py); not cloud-specificPriority: High — search silently serves stale/incorrect content after DB-only divergence, and the documented incremental repair path (
reindex --search) reports success while doing nothing. No data loss, but a correctness/trust issue for search results.Complexity: Medium — the fix is scoped (extend the FTS reconciliation sweep already present for the
--embeddings/deleted-entity path to also run for--search, purge orphanedsearch_indexrows by id, and add a regression test for a seeded DB↔FTS divergence on an unchanged file). No architectural changes required, but touches the_purge_stale_search_rows/search_service.index_entity_datainteraction so needs careful testing to avoid affecting the unrelated--fullpath.Verification: Read
select_project_index_target_filesinsrc/basic_memory/indexing/project_index_coordinator.py:229-242— it does build its target set solely fromchange_report.new_files | modified_files, which come from file-checksum diffing (change_detector.py), so a DB-only divergence (no file change) indeed produces an empty target set andreindex --searchno-ops. Root cause as described in the issue is accurate.Duplicate check: No existing open or closed issue duplicates this. #1214 is a related companion issue (generation-versioned observation persistence / the field report that surfaced this), not a duplicate — already correctly cross-referenced in the issue body.
Labels applied:
bug, no cloud-specific component.