Skip to content

reindex --search cannot repair DB-only search index divergence #1226

Closed

Description

@phernandez

Problem

basic-memory reindex --search cannot repair database-only divergence: it selects files to
index purely from file-content checksum changes, so when the observation/relation/
search_index tables disagree with an unchanged markdown file, the incremental pass selects
zero files and reports success while search keeps serving stale rows.

Field report on #1214 (SQLite, concurrent MCP clients): after trimming duplicate
observation rows, search_index still held 3886 rows of type='observation' against 2146
observation rows. reindex --search reported Indexing files... 0/0 files and changed
nothing; until reindex --full --search ran, search answered with content no longer in the
store.

Root cause

select_project_index_target_files (src/basic_memory/indexing/project_index_coordinator.py:229-242)
builds its target set from change_report.new_files | modified_files, which
change_detector.py derives purely from file checksums (load_indexed_file_checksums).
DB-only divergence leaves every file in unchanged_files, so nothing reaches
search_service.index_entity_data — the only code path that deletes and rewrites an
entity's observation search rows. _purge_stale_search_rows only handles rows for deleted
entities, and only runs in the --embeddings path.

Proposed direction

Add a DB↔FTS reconciliation sweep to the search reindex path (extend
_purge_stale_search_rows or a sibling): purge search_index rows whose id no longer
exists in the owning table (type='observation' → observation.id, type='relation' →
relation.id), and optionally detect entities whose row counts disagree and re-enqueue just
those entities, so --search without --full converges without a full rebuild.

The one-time damage from the observation duplication bug is handled by the #1214 migration's
FTS orphan purge; this issue is about making incremental reindex --search able to repair
such divergence in general.

Acceptance criteria

  • reindex --search (no --full) removes search_index rows of type='observation'/type='relation' whose backing row no longer exists.
  • A seeded DB↔FTS divergence on an unchanged file converges via reindex --search alone (regression test).
  • No full-file re-parse is required for the purge path; --full behavior unchanged.

Refs: #1214 (field report), #1213.

Activity

  1. claude commented on Aug 11, 2026

    @claude
    Contributor

    Claude finished @phernandez's task in 41s —— View job


    Triage

    Type: Bug (confirmed — bug label already applied)
    Component: Database/sync — reindex --search indexing pipeline (src/basic_memory/indexing/project_index_coordinator.py); not cloud-specific
    Priority: High — search silently serves stale/incorrect content after DB-only divergence, and the documented incremental repair path (reindex --search) reports success while doing nothing. No data loss, but a correctness/trust issue for search results.
    Complexity: Medium — the fix is scoped (extend the FTS reconciliation sweep already present for the --embeddings/deleted-entity path to also run for --search, purge orphaned search_index rows by id, and add a regression test for a seeded DB↔FTS divergence on an unchanged file). No architectural changes required, but touches the _purge_stale_search_rows / search_service.index_entity_data interaction so needs careful testing to avoid affecting the unrelated --full path.

    Verification: Read select_project_index_target_files in src/basic_memory/indexing/project_index_coordinator.py:229-242 — it does build its target set solely from change_report.new_files | modified_files, which come from file-checksum diffing (change_detector.py), so a DB-only divergence (no file change) indeed produces an empty target set and reindex --search no-ops. Root cause as described in the issue is accurate.

    Duplicate check: No existing open or closed issue duplicates this. #1214 is a related companion issue (generation-versioned observation persistence / the field report that surfaced this), not a duplicate — already correctly cross-referenced in the issue body.

    Labels applied: bug, no cloud-specific component.

  2. added this to the v0.23 milestone on Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions