You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
server(openai): implement POST /v1/embeddings — needs a real pooling path in Engine, not a shim #451
POST /v1/embeddings is absent, and unlike the gaps in #450 this is a feature, not a shim: there is no pooling path in DotLLM.Engine to build
it on. A grep for embeddings across src/DotLLM.Server and src/DotLLM.Engine returns only an unrelated hit in RateLimitMiddleware.
Scope
A forward path that returns the hidden state instead of logits, and stops
before lm_head.
Pooling strategies: last, mean, cls — GGUF metadata
(*.pooling_type) selects the default where present.
Batch input: string, array of strings, and pre-tokenised int arrays.
usage.prompt_tokens accounting.
Notes
Backend coverage should start CPU-only and be explicit about it; a Vulkan
path is a follow-on, not a blocker.
Correctness needs an external anchor (llama.cpp llama-embedding on the same
GGUF), not just self-consistency — cosine similarity against a reference
run, per the project's oracle convention.
Acceptance criteria
client.embeddings.create() from the official OpenAI SDK returns a
well-formed response.
Cosine similarity vs a llama.cpp reference on the same model/quant is within
a stated tolerance, with the tolerance justified.
Implemented on branch issue/451-embeddings (5 commits, not pushed).
Backend coverage is CPU only, deliberately and explicitly. The pooled hidden state comes from a new IEmbeddingModel.ForwardHidden, which only the CPU TransformerModel implements. IEmbeddingModel is a separate interface from IModel precisely so the Vulkan and CUDA models do not silently acquire an unvalidated embeddings path: when one of those is loaded, POST /v1/embeddings returns 501 Not Implemented naming the model type. A GPU path is a follow-on, not a blocker. This is stated in docs/SERVER.md, in the endpoint's XML docs, and on roadmap step 34.
What landed
ForwardHidden stops at llama.cpp's result_norm — post final output norm, pre LM head (src/models/llama.cpp: res->t_embd = cur;), which is the tensor llm_graph_context::build_pooling operates on.
EmbeddingPooler: last / mean / cls, semantics taken from build_inp_mean (uniform 1/n) and build_inp_cls (row select); L2 normalisation matching common_embd_normalize step for step.
GGUF {arch}.pooling_type is read into ModelConfig.PoolingType using llama.cpp's own enum values. Precedence: explicit request → checkpoint declaration → last. The final fallback is a documented deviation from llama.cpp's NONE default, which is not representable in an OpenAI response.
External anchor. Reference vectors captured from llama.cpp llama-server --embeddings (b9672, -ngl 0) on the same GGUF — llama-embedding is not in the prebuilt release package on this box, but it is the same library and the same build_pooling graph. Committed with full provenance plus the capture script. Cosine vs that reference, SmolLM2-135M-Instruct Q8_0, two inputs:
pooling
input 0
input 1
last
0.99985868
0.99975375
mean
0.99946456
0.99993534
cls
0.99998587
0.99998502
Gate is cosine ≥ 0.9974, placed at the log-midpoint of the measured gap to the broken forms (worst broken form 0.98717), giving ~4.9× headroom each side. Every broken form — wrong pooling mode, off-by-one row, off-by-one token count, wrong graph point — is asserted to fall below it, and six deliberate mutations were run to confirm the tests actually fail against them. Thread count is ruled out as a source of drift: the same six cosines come out bit-identical single-threaded.
One thing worth flagging beyond the issue's scope. The server request gate alone did not make this safe. ContinuousBatchSchedulerService's run loop drives forward passes on the same model outside that gate by design, and it is enabled by default for paged-KV CPU models. Removing the fix and running a /v1/completions generation alongside /v1/embeddingsaborts the process (CountdownEvent ... below zero out of the shared ComputeThreadPool). Fixed by adding ContinuousBatchSchedulerService.AcquireModelAsync() — the run loop takes the same lock around each Step, so an out-of-band forward interleaves between steps. Cost is one uncontended semaphore per forward pass.
Activity
jamesburton commentedon Sep 21, 2026
Implemented on branch
issue/451-embeddings(5 commits, not pushed).Backend coverage is CPU only, deliberately and explicitly. The pooled hidden state comes from a new
IEmbeddingModel.ForwardHidden, which only the CPUTransformerModelimplements.IEmbeddingModelis a separate interface fromIModelprecisely so the Vulkan and CUDA models do not silently acquire an unvalidated embeddings path: when one of those is loaded,POST /v1/embeddingsreturns 501 Not Implemented naming the model type. A GPU path is a follow-on, not a blocker. This is stated indocs/SERVER.md, in the endpoint's XML docs, and on roadmap step 34.What landed
ForwardHiddenstops at llama.cpp'sresult_norm— post final output norm, pre LM head (src/models/llama.cpp:res->t_embd = cur;), which is the tensorllm_graph_context::build_poolingoperates on.EmbeddingPooler: last / mean / cls, semantics taken frombuild_inp_mean(uniform 1/n) andbuild_inp_cls(row select); L2 normalisation matchingcommon_embd_normalizestep for step.{arch}.pooling_typeis read intoModelConfig.PoolingTypeusing llama.cpp's own enum values. Precedence: explicit request → checkpoint declaration →last. The final fallback is a documented deviation from llama.cpp'sNONEdefault, which is not representable in an OpenAI response.inputas string / string[] / int[] / int[][],encoding_format: float|base64,usage.prompt_tokens.External anchor. Reference vectors captured from llama.cpp
llama-server --embeddings(b9672,-ngl 0) on the same GGUF —llama-embeddingis not in the prebuilt release package on this box, but it is the same library and the samebuild_poolinggraph. Committed with full provenance plus the capture script. Cosine vs that reference, SmolLM2-135M-Instruct Q8_0, two inputs:Gate is cosine ≥ 0.9974, placed at the log-midpoint of the measured gap to the broken forms (worst broken form 0.98717), giving ~4.9× headroom each side. Every broken form — wrong pooling mode, off-by-one row, off-by-one token count, wrong graph point — is asserted to fall below it, and six deliberate mutations were run to confirm the tests actually fail against them. Thread count is ruled out as a source of drift: the same six cosines come out bit-identical single-threaded.
One thing worth flagging beyond the issue's scope. The server request gate alone did not make this safe.
ContinuousBatchSchedulerService's run loop drives forward passes on the same model outside that gate by design, and it is enabled by default for paged-KV CPU models. Removing the fix and running a/v1/completionsgeneration alongside/v1/embeddingsaborts the process (CountdownEvent ... below zeroout of the sharedComputeThreadPool). Fixed by addingContinuousBatchSchedulerService.AcquireModelAsync()— the run loop takes the same lock around eachStep, so an out-of-band forward interleaves between steps. Cost is one uncontended semaphore per forward pass.feat(server): POST /v1/embeddings with a real pooling path (#451)
fix(server): meter by exemption, not by allowlist (#452)
test(server): official OpenAI + Anthropic SDK conformance matrix (#453)
test: unit + HTTP coverage for the embeddings path (#451)
docs: document POST /v1/embeddings and its CPU-only scope (#451)
fix(server): guard the embeddings forward against the scheduler loop (#…
test: re-centre the parity gate and rule out thread-count drift (#451)
fix(server): reconcile ErrorResponse call sites across the merged bra…
merge: OpenAI + Anthropic compatibility, embeddings, management API a…
fix(server): take the scheduler lease inside ExecuteAsync, not per ca…