Indexing leaks ~9 MB anon-rss per chunk in single process; OOM-kills mid-corpus on 15 GB host

[rayrage213]

Bug: indexing a multi-MB markdown corpus with the ONNX provider walks anon-rss to ~11.5 GB and the kernel OOM-kills the process partway through. Memory growth is linear in chunks indexed within a single process; per-process restart releases memory normally, so the leak is in long-lived state inside MemSearch/pymilvus/milvus-lite interaction rather than ONNX itself.

The naïve "split work into separate memsearch index <file> calls" workaround is sabotaged by MemSearch.index()'s implicit stale-source cleanup (delete_by_source for any source not in the just-passed paths) — per-file CLI calls treat each other's files as deleted and wipe them.

Environment

• memsearch 0.4.1 (pip)
• pymilvus 2.5.x (whatever 0.4.1 pulls in), milvus-lite from same constraint
• onnxruntime 1.24.4, model gpahal/bge-m3-onnx-int8 (the plugin default)
• Python 3.13.x in a venv
• Linux 6.12 / 4-vCPU AMD EPYC 9354P / 15 GB RAM, no swap
• Corpus: ~518 markdown files, ~7 MB total, ~3000 expected chunks at max_chunk_size=4000 chars

Reproduction

memsearch index --batch-size 4 --max-chunk-size 4000 --force <full-corpus-paths>

Equivalent in code: a single MemSearch.index() call against the full corpus.

Observed

Run	--batch-size	--max-chunk-size	ONNX arena	Chunks before OOM	Peak anon-rss	Peak total-vm
1	8	4000	enabled	~80	11.85 GB	17.97 GB
2	4	4000	enabled	~628	11.55 GB	18.77 GB
3	4	4000	disabled (SessionOptions.enable_cpu_mem_arena=False)	~628	11.53 GB	12.94 GB
4	4	2000	enabled	~1186	11.55 GB	18.77 GB

Kernel log:

kernel: memsearch invoked oom-killer: gfp_mask=0x140cca, order=0, oom_score_adj=0
kernel: Out of memory: Killed process <pid> (memsearch) total-vm:18756488kB, anon-rss:11587564kB,
        file-rss:156kB, shmem-rss:0kB, UID:1001 pgtables:23552kB oom_score_adj:0

Per-chunk leak ≈ (peak_anon_rss − baseline) / chunks_processed:

• run 2: ≈ 16 MB/chunk
• run 4: ≈ 8.4 MB/chunk

Per-chunk leak halves with chunk size halved → some of it is working-set memory (tokenizer / batch pad / activations). But disabling the ONNX CPU mem arena (run 3) does not materially reduce peak anon-rss — only total-vm drops by ~6 GB. So the dominant leaked bytes are not in the ONNX session arena.

Per-file probe (instrumented index_file loop in one process)

Indexed Log/DECISIONS.md (934 KB → 541 chunks) as the first file:

start RSS = 1153 MB
[1/518] +541 chunks  rss = 6064 MB  Log/DECISIONS.md
[2/518] + 39 chunks  rss = 6064 MB  Log/OC-RETIREMENT.md

A single big file alone drives RSS up by ~5 GB / 541 chunks ≈ 9 MB/chunk persistent. The next, smaller file does not add memory — RSS plateaus until another large file appears, then climbs again. So peak anon-rss tracks "max chunks-in-flight cumulative" plus a per-chunk persistent residue in the long-lived MilvusClient state.

Naïve workaround that doesn't work

Calling memsearch index <single-file> per file in a shell loop. Each call's MemSearch.index() runs:

# memsearch/core.py — at end of index()
indexed_sources = self._store.indexed_sources()
for source in indexed_sources:
    if source not in active_sources:
        self._store.delete_by_source(source)

active_sources is built from the paths just passed to that invocation, so per-file calls treat the rest of the corpus as deleted. After a 6-file shell loop the DB only contains the last file's chunks (39 of OC-RETIREMENT.md).

Two paths to fix this from the user side:

1. Use MemSearch.index_file(path) directly (no stale-source cleanup) and shard the corpus into byte-budget batches with subprocess restarts.
2. Pass the full corpus path list every call (so cleanup is a no-op) and rely on chunk-hash dedup to skip already-indexed chunks. Doesn't help — the first call still has to embed the whole corpus.

We went with (1).

Working workaround (open to PRing back)

• A small Python helper that calls index_file per path (no destructive cleanup): https://github.com//blob/.../recall-index-files.py
• A bash batcher that sorts files biggest-first, packs into 1 MB / 64-file batches, and runs each batch in its own subprocess so the heap resets between batches: https://github.com//blob/.../recall-reindex.sh

Throughput is unaffected: total ≈ same as a single in-process run would be if it didn't OOM, because per-batch model load is ~5 s and the rest is dominated by ONNX inference time.

Suggested upstream fixes

1. Document or eliminate the implicit stale-source cleanup in MemSearch.index() (or split it into an explicit prune_deleted_sources() API). The current behavior is surprising for callers who shard their corpus across multiple invocations.
2. Investigate the per-chunk persistent leak in long-lived MemSearch instances. Best guess based on this evidence: pymilvus's MilvusClient retains references to inserted records (or grpc / arrow buffers) across upsert calls; smaller chunks halve the per-chunk leak which is consistent with payload retention rather than schema overhead. Worth instrumenting with tracemalloc snapshots before / after each _embed_and_store call on a long corpus.
3. Consider periodic gc.collect() + malloc_trim(0) between files as a stopgap. (Only helps if the leak is reachable garbage, not strong refs.)

Happy to test patches against the reproduction corpus if useful.

[zc277584121]

Thanks for the detailed report. The data here is useful, especially the single-process vs subprocess behavior and the ONNX arena comparison.

To narrow down where the retained memory is coming from, could you share two more comparisons if they are easy to run?

1. Does the same corpus reproduce with a non-ONNX embedding provider, for example OpenAI or another remote embedding provider, while still using Milvus Lite?
2. Does the same ONNX setup reproduce when the store is a remote/Docker Milvus instance instead of Milvus Lite?

The current indexing path embeds a whole file at a time: it builds the cleaned text list, accumulates all batch embeddings, builds one records list, and then sends that whole file's records to MilvusClient.upsert(). For a 541-chunk file, the Python-side embedding list alone should not explain multi-GB RSS, so the distinction between ONNX, Milvus Lite, and the upsert/native-buffer path would be very helpful.

If possible, it would also help to log RSS at these points for the large file case:

• after chunking
• after embedding returns
• after record construction
• after upsert() returns
• after gc.collect()

That should tell us whether the growth is mostly before upsert, during upsert, or retained after upsert.

[rayrage213]

Thanks for the quick reply. Honest answer first: we ended up rewriting our recall layer on sqlite-vec + fastembed after this incident, so I'd be reinstalling memsearch + milvus-lite specifically to debug it on the same 15 GB / no-swap host that OOM'd us originally — and our budget for that isn't there right now. Sorry I can't run the two A/B swaps directly.

What I can offer is a few additional observations from the partial-completion run that didn't make it into the original report, in case any of them narrow things.

Subprocess restart fully releases the leaked RSS

Our working workaround is a bash batcher that sorts files biggest-first, packs them into ~1 MB / 64-file batches, and runs each batch as its own python -c "from memsearch import MemSearch; ..." subprocess. Between batches the Python heap resets cleanly — pgrep -x milvus goes to zero, then the next subprocess starts from baseline. So whatever's accumulating is reachable from in-process state — not the storage files, not a stray milvus-lite child outliving the parent.

Cold-backfill batch run that got us 6,123 chunks / 504 of 520 files

(--batch-size 4 --max-chunk-size 4000, ONNX gpahal/bge-m3-onnx-int8, biggest-first batching)

Batch	Files	KB	Wall	Chunks added	Result
1	1	912	1334 s	541	OK
2	2	906	58 s	0	rc=137
3	12	1006	23 s	0	rc=137
4	39	1013	1478 s	979	OK
5	64	1023	1142 s	1068	OK
6–12	(progressively smaller files)	...	...	3535	all OK

Batch 1 is DECISIONS.md alone (the 912 KB / 541-chunk file from the original report). Batch 2 is the next two largest files (~450 KB each). Batch 3 is 12 medium files. Both batch 2 and batch 3 SIGKILL within 60 s — after batch 1 has already landed 541 chunks into the DB. Batches 6–12, with progressively smaller files but a DB that's now holding several thousand chunks, all complete fine.

I don't have a clean isolation between "load-time RSS on opening a populated DB" and "in-flight working set during index_file()" — both scale with chunk count. But the pattern (small files succeed even with a heavily populated DB; medium files OOM with only 541 chunks already there) is more consistent with most of the cost being per-call working set than per-open populated-DB load.

enable_cpu_mem_arena=False is not just neutral

Disabling the ONNX CPU mem arena (run 3 in the original report) didn't reduce peak anon-rss, but it roughly doubled wall-clock per batch at batch_size=4. We removed the monkey-patch on that basis. Worth flagging since the wall-clock penalty isn't obvious from a memory-only A/B.

What I can't give you

• Non-ONNX embedding provider + milvus-lite — no remote embedding budget wired up
• ONNX + Docker Milvus — would mean re-staging the stack we no longer use

Cheap independent localization you could run

If the populated-DB-on-open path turns out to matter at all, the cheapest version of your "checkpoint RSS" idea is to instantiate MilvusClient(uri="…/data.db") against any milvus-lite DB that already holds ~1000 chunks, and log RSS immediately before vs. immediately after the constructor — no embedding work needed at all. If RSS jumps GB-scale just at open, that's the smoking gun. If it stays flat, the bytes are accumulating somewhere on the upsert/native-buffer path as you suspect.

Happy to answer follow-ups on the data we already have. And thanks for engaging on this — it would have been an easy issue to ignore given we'd already moved on.