[Bug]: While creating chroma client - getting error - no such tokenizer: trigram

[ashu-accolitedigital]

What happened?

When trying to create chroma db client, running into the following error. This error occurs both on google colab

chroma_client = chromadb.PersistentClient(path="/content/chromadb")

Versions

Here is my colab environment-

Python implementation: CPython
Python version : 3.10.12
IPython version : 7.34.0

chromadb: 0.4.0

sqlite3.sqlite_version => 3.31.1

Relevant log output

---------------------------------------------------------------------------
OperationalError                          Traceback (most recent call last)
[<ipython-input-67-2adad2a0c4f4>](https://localhost:8080/#) in <cell line: 1>()
----> 1 chroma_client = chromadb.PersistentClient(path="/content/chromadb")

6 frames
[/usr/local/lib/python3.10/dist-packages/chromadb/db/impl/sqlite.py](https://localhost:8080/#) in apply_migration(self, cur, migration)
    207     @override
    208     def apply_migration(self, cur: base.Cursor, migration: Migration) -> None:
--> 209         cur.executescript(migration["sql"])
    210         cur.execute(
    211             """

OperationalError: no such tokenizer: trigram

[Russell-Pollari]

Duplicate of #837

You'll need a new version of SQLite. I think trigram tokenizer was introduced in 3.34.0

[HammadB]

Yeah as mentioned in #837, unfortunately colab ships with an out of date version of sqlite so you will have to compile your own. We have an example here

https://colab.research.google.com/drive/1QEzFyqnoFxq7LUGyP1vzR4iLt9PpCDXv?usp=sharing

[HammadB]

Hi folks, we've landed #841 and shipped 0.4.1 which reverts using the trigram tokenizer. While this will degrade search quality, we think preserving the colab experience as working is critical and will add an optional path to using more sophisticated tokenizers in the future.