Optimizer store never recovers after a transient embedding failure at startup (no such table: llm_capabilities)

[Tanguille]

Running vmcp v0.40.1 via the Kubernetes operator with the optimizer enabled (embeddingProvider: openai pointing at a litellm proxy in front of a local embedding model), I hit a failure mode where the optimizer store never recovers from a transient embedding outage.

My unified vMCP pod happened to restart during an infra outage, while the embedding backend was still down. The first session that connected tried to build the optimizer and failed on the embedding call, which is fair enough:

build session optimizer: failed to upsert tools into store:
  failed to generate embeddings: OpenAI returned status 500: ... Connection error.. Model Group=vmcp-embedding

The problem is that once the embedder came back, the vMCP never recovered. From that point on, every session build and every find_tool failed with:

build session optimizer: failed to upsert tools into store:
  failed to prepare statement: SQL logic error: no such table: llm_capabilities (1)

find_tool failed: tool search failed:
  FTS5 query failed: SQL logic error: no such table: llm_capabilities_fts (1)

It stayed like that for ~14 hours (about 138 errors) with the pod sitting Ready the whole time — /health and /readyz green, all backends healthy, so nothing surfaced it. Deleting the pod, with the embedder healthy this time, rebuilt the store cleanly and everything worked again.

So it looks like the store schema (llm_capabilities, plus the llm_capabilities_fts FTS5 table) is created lazily on the first upsert, and that upsert generates embeddings before it writes. When the embedding call fails, the tables never get created, and there's no retry — the in-memory per-pod store is stuck without a schema until the process restarts.

Would it make sense to create the schema at store/optimizer init, before and independent of any embedding call? Or make that first upsert retry, so a transient embedder blip can't leave the store permanently schemaless. As it stands, any restart that happens to coincide with the embedding backend being down (node reboot, infra blip, embedder rollout) can silently take the gateway out until someone notices there are no tools and restarts the pod.

Might be related to #5847 (same optimizer/Serve path), though that one is about re-embedding cost rather than a hard schema failure.

[aponcedeleonch]

Thanks for reporting this!

The failure is real, but the cause isn't quite what it looks like from the outside.

What's actually happening

The schema isn't created lazily. newSQLiteToolStore executes the whole of schema.sql eagerly in the constructor, and it did in v0.40.1 too. So both tables exist before any embedding call happens.

The problem is connection lifetime. The store runs on file:memdb?mode=memory&cache=shared, and a shared-cache in-memory SQLite database only exists while at least one connection to it is open. If the database/sql pool ever drops to zero connections, SQLite throws away the entire database: table, FTS5 index, triggers. Nothing recreates it, because the schema only ever runs in the constructor. That's why it stayed broken for 14 hours and why deleting the pod fixed it.

The fix

PR: #6157

Pin a dedicated connection for the store's lifetime. It's acquired with a background context and never used to run statements, so it can't be interrupted and therefore can't be discarded. That keeps the database alive regardless of what happens to the rest of the pool.

[aponcedeleonch]

@Tanguille Fixed it! Thanks for reporting