[Bug]: Correctness bug in log purge - causes loss of embeddings data

[tazarov]

What happened?

0.5.7

To reproduce in 0.5.7

pip install chromadb==0.5.7

import chromadb
import uuid


client = chromadb.PersistentClient(path="2912")

ids = [str(uuid.uuid4()) for _ in range(5202)]
docs = [f"Document {i}" for i in range(5202)]
embeddings = [[0.1, 0.2, 0.3] for _ in range(5202)]

col = client.get_or_create_collection("test_collection")

col.add(ids=ids, documents=docs, embeddings=embeddings)


ids1 = [str(uuid.uuid4()) for _ in range(2202)]
docs1 = [f"Document {i}" for i in range(2202)]
embeddings1 = [[0.1, 0.2, 0.3] for _ in range(2202)]

col1 = client.get_or_create_collection("test_collection1")

col1.add(ids=ids1, documents=docs1, embeddings=embeddings1)

Restart the notebook/server/session:

import chromadb
client = chromadb.PersistentClient(path="2912")
col = client.get_or_create_collection("test_collection")
res= col.get(include=["documents", "embeddings"])
assert len(res['embeddings']) == 5202

results in:

IndexError                                Traceback (most recent call last)
[<ipython-input-1-466544d3b9d7>](https://jpcubrfly0m-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20241007-060114_RC01_683255037#) in <cell line: 4>()
      2 client = chromadb.PersistentClient(path="2912")
      3 col = client.get_or_create_collection("test_collection")
----> 4 res= col.get(include=["documents", "embeddings"])
      5 assert len(res['ids']) == 5202

5 frames
[/usr/local/lib/python3.10/dist-packages/chromadb/segment/impl/vector/local_persistent_hnsw.py](https://jpcubrfly0m-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20241007-060114_RC01_683255037#) in get_vectors(self, ids)
    388             for label, vector in zip(hnsw_labels, vectors):
    389                 id = self._label_to_id[label]
--> 390                 results[id_to_index[id]] = VectorEmbeddingRecord(
    391                     id=id, embedding=vector
    392                 )

IndexError: list assignment index out of range

0.5.12

Here, the issue exhibits somewhat differently (this is another bug that needs fixing in a separate Issue/PR). While one does not get the IndexError: list assignment index out of range, asserting the expected embedding count points to the same issue:

pip install chromadb==0.5.12

import chromadb
import uuid


client = chromadb.PersistentClient(path="2912")

ids = [str(uuid.uuid4()) for _ in range(5202)]
docs = [f"Document {i}" for i in range(5202)]
embeddings = [[0.1, 0.2, 0.3] for _ in range(5202)]

col = client.get_or_create_collection("test_collection")

col.add(ids=ids, documents=docs, embeddings=embeddings)


ids1 = [str(uuid.uuid4()) for _ in range(2202)]
docs1 = [f"Document {i}" for i in range(2202)]
embeddings1 = [[0.1, 0.2, 0.3] for _ in range(2202)]

col1 = client.get_or_create_collection("test_collection1")

col1.add(ids=ids1, documents=docs1, embeddings=embeddings1)

Restart the notebook/server/session:

import chromadb
client = chromadb.PersistentClient(path="2912")
col = client.get_or_create_collection("test_collection")
res= col.get(include=["documents", "embeddings"])
print("Actual len: ",len(res['embeddings']))
assert len(res['embeddings']) == 5202

Results in:

Actual len:  5000
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
[<ipython-input-5-7e030716345f>](https://glksq6xpilu-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20241007-060114_RC01_683255037#) in <cell line: 6>()
      4 res= col.get(include=["documents", "embeddings"])
      5 print("Actual len: ",len(res['embeddings']))
----> 6 assert len(res['embeddings']) == 5202

AssertionError:

Analysis

The issue comes from how we purge the embeddings queue:

chroma/chromadb/db/mixins/embeddings_queue.py

Lines 162 to 171 in cb88db2

	t = Table("embeddings_queue")
	q = (
	self.querybuilder()
	.from_(t)
	.where(t.seq_id < ParameterValue(min_seq_id))
	.delete()
	)
	sql, params = get_sql(q, self.parameter_format())
	cur.execute(sql, params)

The above code results in an indiscriminate purge of the embedding queue up to the given collection's min_seq_id (usually the HNSW seq_id). This is logically incorrect in a multi-collection scenarios where other collections may have entries in the embeddings queue with sequences < the purged collection min_seq_id

Impact

The impact of this is quite high and easily reproducible for deployments with more than one collection which are on versions > 0.5.7.

The result of the impact is that while documents and metadata are present the embeddings which have not been committed to HNSW (aka sync_threshold not reached) are lost.

Versions

Chroma version 0.5.7-0.5.12 (single-node and persistent)

Relevant log output

No response