OpenAI-Compatible embedding dimension fail for codebase indexing

[dflatline]

App Version

3.19.5

API Provider

Ollama via OpenAI-Compatible Embedding API

Model Used

snowflake-arctic-embed2

🔁 Steps to Reproduce

Holy smokes there's so many bugs with the Codebase Indexing support I am not even sure where to start.

First of all, the embedding provider config does not reliably save itself and resets both the provider choice and random config param values, so if you try to reproduce this, make sure it actually saved your settings by checking back to it a few times...

I finally did get codebase indexing working with a fix to the OpenAI-Compatible provider. I was previously blocked by this: #1935, which is why I am using OpenAI-Compatible for ollama instead of native ollama support.

Steps to reproduce this current bug:
0. Run ollama pull snowflake-arctic-embed2

1. Run docker run -p 6333:6333 qdrant/qdrant
2. Enable Codebase Indexing checkbox in Roo Experimental Settings (flask icon)
3. Set OpenAI-Compatible as embeddings provider
4. Set Base URL to http://127.0.0.1:11434/v1
5. Set API Key to whatever
6. Set embedding model to snowflake-arctic-embed2
7. Set embeddings dimension to 1024
8. Leave qdrant url as http://localhost:6333.

Click Save and Done. Restart vscode, make sure these settings applied. Set them again. Restart again.

Do this a few times until all settings actually save. They will not save properly the first time.

Once they finally have saved such that they are the same after a restart, click "index"

Watch your qdrant docker container logs. it will have the error:
WARN collection::shards::replica_set::update: Failed to update shard ws-ee207033c79ae4df:0 on peer 4059754735917558, error: Wrong input: Vector dimension error: expected dim: 1024, got 256

This is because the OpenAI typescript package has some deep dark code that misparses the embeddings array. Maybe a mismatch with OpenAI's actual API's encoding mechanisms? Probably sabotage, though 😜.

I have a patch here: https://github.com/dflatline/Roo-Code/tree/embed_dimfix2.

As soon as the bot finishes giving me useless advice, I will post an MR and instructions on getting it working with ollama (there's also a batch size problem to hack around, which also degrades embedding quality and causes coverage loss: ollama/ollama#6262).

💥 Outcome Summary

Expected useless advice from the auto-reply bot. Got useless advice from the useless auto-reply bot.

I have a fix though! I finally got codebase indexing working! Will provide instructions and a patch after the bot spams us with nonsense.

📄 Relevant Logs or Errors (Optional)

The qdrant log to watch for is:
`WARN collection::shards::replica_set::update: Failed to update shard ws-ee207033c79ae4df:0 on peer 4059754735917558, error: Wrong input: Vector dimension error: expected dim: 1024, got 256`

But only if you specify an embedding dimension. If you don't I bet the OpenAI sabotage succeeds and you see nothing; you just store truncated embeddings (which are less accurate, of course, making you want to pay for their official API and give them a copy of all your data).

[dflatline]

Hrmm even with my fix from #4463 running, and with ollama launched as OLLAMA_NUM_PARALLEL=1 OLLAMA_MAX_QUEUE=1000000 ollama serve as per ollama/ollama#6262, the accuracy of returned results is still seems poor..

I am watching the data flow to and from ollama and qdrant with wireshark, and it does look like everything is being embedded and stored properly, though I am not deeply familiar enough with qdrant to really know if the store and search is being done correctly/optimally. :/

[dflatline]

Hrmm even with my fix from #4463 running, and with ollama launched as OLLAMA_NUM_PARALLEL=1 OLLAMA_MAX_QUEUE=1000000 ollama serve as per ollama/ollama#6262, the accuracy of returned results is still seems poor..

Looking at debug logs of ollama, I still see "decode: cannot decode batches with this context (use llama_encode() instead)" in this case, so ollama/ollama#6262 is still the cause of my accuracy issues, even with the recommended env vars.

LM Studio is significantly better here. With the exact same embedding model (text-embedding-snowflake-arctic-embed-l-v2.0@q6_k), with this fix, the Code Index feature returns somewhat relevant results.

I have also since learned that most of the open-weight embedding frontier is not that great at code at all, so nomic released nomic-embed-code: https://huggingface.co/nomic-ai/nomic-embed-code https://arxiv.org/abs/2412.01007.

Unfortunately, that can't run because of lmstudio-ai/lms#205. (nomic-embed-code is basically an intermediate layer output of a fine-tuned 7B LLM). Next up is llama.cpp: https://huggingface.co/nomic-ai/nomic-embed-code-GGUF/discussions/1

I have also verified that without the fix from #4463, LM Studio also experiences this ticket's original embedding truncation sabotage^W bug, and qdrant also emits:
2025-06-09T02:33:45.736761Z WARN collection::shards::replica_set::update: Failed to update shard ws-ee207033c79ae4df:0 on peer 7866600142623554, error: Wrong input: Vector dimension error: expected dim: 1024, got 256

[dflatline]

To reproduce this bug with llama.cpp and nomic-embed-code (the new code embedding hotness):

brew install llama.cpp
llama-server -m ~/.cache/lm-studio/models/lmstudio-community/nomic-embed-code-GGUF/nomic-embed-code-Q8_0.gguf --embeddings -c 4096 -ngl 99

The correct embedding dimension setting for nomic-embed-code is 3584 (nomic-embed-code embedding dimension information is literally no where else on the internet. Not even their own research paper 😂).

Also, even though I set a context size of 4096 in llama.cpp, looks like I never get a chunk over 512 bytes. Worth noting that longer chunks are better for retrieval accuracy (and also help the llm think better): https://www.llamaindex.ai/blog/evaluating-the-ideal-chunk-size-for-a-rag-system-using-llamaindex-6207e5d3fec5

(Maybe the chunk size should also be a Codebase Indexing option?).

[oOBigManOo]

@dflatline Iam gettings chunks over 512 bytes which makes me unable to use mxbai-embed-large through llama-server using openai comptaible option
in my case it forces me to install ollama and use it
Chunk size must be configurable in roo code

[sxueck]

I've also encountered a new issue: after configuring the index, it only shows "filewatcher started," and Qdrant shows 0 points
Using the tool for codebase search doesn't produce errors (but also doesn't return any results), which is a very poor experience