Orphaned index generations accumulate when the previous manifest read fails — store grows until the server pegs a core while idle

[KEONHEEJ]

Package: @agentmemory/agentmemory 0.9.28
iii runtime: 0.11.2
Host: macOS 26.5.2, arm64, Node v24.16.0, launchd-managed (not Docker)
Store: file-based KV, ~/data/state_store.db, ~3,000 observations

Summary

IndexPersistence.save() cleans up the previous generation's shards only when it successfully read the previous manifest. When that read fails — which happens exactly when the store is under load — the cleanup is skipped entirely and that generation's shards are orphaned forever: every later save() only ever looks at its own previous, so nothing ever revisits older orphans.

Orphans are monotonically accumulating, and they feed back into the failure that creates them.

In my install this reached 8 index generations / 140 MB of shards, of which only 2 generations (37 MB) were live. The server then sat at 110–150% CPU while completely idle with every HTTP route timing out.

Evidence

Storage breakdown before cleanup (338 MB total):

Key prefix	Size	Kind
mem:index:*	140 MB, 77 files, 8 generations	derived (only 2 gens live)
mem:graph:*	85 MB	derived
mem:audit	42 MB	log
mem:obs:* (2,888 observations)	66 MB	real data

So 79% of the store was derived data, most of it unreachable.

The manifest names the live generations explicitly:

{"data:manifest":    {"generation": "idx_mrzxkql0_d41f7623ee4f", "shards": [...]},
 "vectors:manifest": {"generation": "idx_mrzxkrwf_8fe2c54a3dc8", "shards": [...]}}

The other six idx_* generations were referenced by nothing.

Effect of deleting the orphans + graph + audit (observations untouched):

Metric	Before	After
Idle CPU (no requests in flight)	110–150%	0–3%
Wrapper RSS	1.4–2.7 GB, oscillating	93–250 MB
GET /agentmemory/health	8–25 s, often timeout	6 ms
POST /agentmemory/search	20 s timeout	14 ms
POST /agentmemory/smart-search	44.5 s	0.82 s
Store size	338 MB	109 MB

The server has now been up 45+ hours with no restarts and no degradation.

Root cause

dist/index.mjs, end of IndexPersistence.save():

await this.deleteKey(KV.bm25Index, legacyKey, "legacy_cleanup");
if (previous?.v === 1 && Array.isArray(previous.shards)) {
  const currentShardIds = new Set(shards.map((shard) => `${shard.scope}\0${shard.key}`));
  for (const shard of previous.shards) {
    if (currentShardIds.has(`${shard.scope}\0${shard.key}`)) continue;
    await this.deleteShards([shard], "previous_generation_cleanup");
  }
}

previous comes from reading the manifest at the start of save(). That read fails under load — from my logs:

[agentmemory] warn index persistence: BM25 manifest read failed
  {"scope":"mem:index:bm25","key":"data:manifest","message":"Invocation timeout after 180000ms: state::get"}
[agentmemory] warn index persistence: vector manifest read failed
  {"scope":"mem:index:bm25","key":"vectors:manifest","message":"Invocation timeout after 180000ms: state::get"}

When that happens previous is nullish, the if is skipped, and the shards written by the preceding generation are never deleted by anyone. There is no reconciliation pass that would find them later.

I found zero previous_generation_cleanup audit records against 3 logged manifest-read failures, which is consistent.

Why it compounds

1. Store grows → state::get gets slower
2. Manifest read hits the 180 s invocation timeout
3. Cleanup skipped → one more generation orphaned permanently
4. Store grows further → go to 1

Each failure is permanent, so the floor only ever rises. This also explains why the symptom appears suddenly after weeks of fine operation: it is not linear, it is ratcheting.

A restart makes it worse, because a fresh save() mints yet another generation — so debugging by restarting is actively counterproductive. During one incident I restarted eight times and ended up with eight generations.

Reproduction

Any deployment where a manifest read times out at least once. To force it:

1. Grow the store until state::get on the manifest key approaches the invocation timeout (~3,000 observations with vectors + graph did it here).
2. Trigger an index save.
3. Observe manifest read failed in the log.
4. ls ~/data/state_store.db | grep 'mem%3Aindex%3A' — the previous generation's shards are still present and are not referenced by mem%3Aindex%3Abm25.bin.

Suggested fixes

1. Reconcile instead of relying on previous. When the manifest read fails, don't silently skip. Enumerate the mem:index:bm25 scope and delete any idx_* shard whose generation is not named by the freshly published manifest. The manifest is the authority and it is written before this step, so a scan-based sweep is safe and self-correcting.
2. Add a startup reconciliation pass. One sweep at boot would bound the damage to a single generation regardless of how many past failures occurred.
3. Reconsider the 180 s invocation timeout for manifest reads. Three minutes of a blocked read on the single-threaded runtime is itself an availability problem; a short timeout plus explicit retry would fail faster and more visibly.
4. Surface the skip. manifest read failed is logged at warn, but the consequence (cleanup skipped, generation orphaned) isn't stated. Making it explicit would have saved a lot of diagnosis time.

Happy to open a PR for (1) + (2) if the approach sounds right.

Workaround for anyone hitting this

Stop the server, back the store up, then keep only the generations the manifest names:

# 1. read the live generations
python3 -c "
import re;print('\n'.join(sorted(set(re.findall(r'idx_[a-z0-9]+_[a-f0-9]+',
  open('~/data/state_store.db/mem%3Aindex%3Abm25.bin',encoding='utf8',errors='replace').read())))))"
# 2. delete every mem%3Aindex%3A* shard whose generation is not in that list

⚠️ Do not delete mem%3Aindex%3A* wholesale — the index is not rebuilt automatically. I tried; the server came up healthy and fast, and every query returned 0 results while all 2,888 observations sat there intact. Keeping the manifest plus the two live generations restored search completely.

mem:graph:* and mem:audit are regenerated and can be dropped safely (dropping the 85 MB graph is what took smart-search from 44.5 s to 0.82 s).

Related

• Architecture: Migrate BM25 & Graph Search from In-Memory to SQLite #309 (closed) — in-memory BM25/graph architecture; this is a consequence of the sharded-persistence design that came out of it
• Stopgap for #309: shard persisted BM25/vector snapshots to avoid oversized state values #762 (closed) — introduced the sharded manifest/generation scheme this issue is about
• Idle CPU leak: iii worker threads busy-wait, growing to a full core over ~40h uptime #1093 (open) — idle CPU growth over ~40 h. Looks similar from the outside but is a different layer: that one is iii worker threads busy-waiting and clears on restart, whereas this one is the Node wrapper and survives restarts because the orphans are on disk.

[sla-te]

Hitting this on Linux at about 4x the size you describe, and I think the trigger is broader than a load timeout: a plain systemctl --user stop reproduces the divergence with nothing else running.

agentmemory 0.9.29, Node 24.19, systemd user unit, file-based KV at ~/.agentmemory/data/state_store.db. 257 sessions, 11165 observations, 4429 memories.

Storage before I touched anything (1.2 GB total):

mem:index:*   855 MB  677 files  14 generations   derived (2 live)
mem:graph:*   112 MB                              derived
mem:obs:*      79 MB   201 files                  real data
mem:memories   22.5 MB                            real data
mem:audit      18.8 MB                            log

So 79% derived here too, and the index alone is ~38x the size of the memories it indexes. Eight vector generations and six BM25 generations, oldest dated 2026-08-01.

The part I did not expect: I stopped the daemon cleanly to apply your workaround, and the shutdown itself made the store inconsistent.

23:25:53  manifest written, names idx_mt2154i9 (bm25, 46 shards) + idx_mt2155we (vectors, 125 shards)
23:58:14  shutdown flush writes idx_mt22aq3p (bm25, 22 shards) + idx_mt22ard5 (vectors, 49 shards)
          and deletes part of the previous generation
          manifest is NOT rewritten - mtime stays 23:25:53

After the stop, disk holds only 20 of the 46 shards the manifest names for idx_mt2154i9, and 69 of 125 for idx_mt2155we. The complete newer pair is named by nothing. Restarting did not fix it either: health and search come back fine (search returns correct results in ~2 s, so the in-memory index is good), but the manifest is still stamped 23:25:53 and still points at the incomplete generation. Presumably related to #1180, since nothing schedules a persist after boot.

Which means your workaround is not safe on this state, and that is worth a warning in the issue: keeping only the generations the manifest names would have deleted the one complete index and kept a half-deleted one. I stopped before running the delete.

No idle-CPU symptom yet - iii sits at 2.5% idle - but RSS was 5.0 GB before the restart and 3.1 GB after, which looks like the same ramp you described before things fell over.

Two things that would help from outside:

1. Is there any supported way to force a consistent manifest write, so orphans can be identified safely? rebuildIndex only runs on an empty index per No way to rebuild the search index without restarting: rebuildIndex only runs when the index is empty #1179, so I could not find one.
2. If the shutdown path writes shards without publishing a manifest, then your suggested fix (2), a startup reconciliation sweep, would not be enough on its own - boot would reconcile against a manifest that is already behind. A sweep after the shutdown flush, or publishing the manifest before the flush returns, looks like it needs to happen too.

Happy to run any diagnostic on this store, it is a decent worst case.