Skip to content

server(openai): implement POST /v1/embeddings — needs a real pooling path in Engine, not a shim #451

Open

Description

@jamesburton

POST /v1/embeddings is absent, and unlike the gaps in #450 this is a
feature, not a shim: there is no pooling path in DotLLM.Engine to build
it on. A grep for embeddings across src/DotLLM.Server and
src/DotLLM.Engine returns only an unrelated hit in RateLimitMiddleware.

Scope

  • A forward path that returns the hidden state instead of logits, and stops
    before lm_head.
  • Pooling strategies: last, mean, cls — GGUF metadata
    (*.pooling_type) selects the default where present.
  • L2 normalisation, encoding_format: float | base64.
  • Batch input: string, array of strings, and pre-tokenised int arrays.
  • usage.prompt_tokens accounting.

Notes

  • Backend coverage should start CPU-only and be explicit about it; a Vulkan
    path is a follow-on, not a blocker.
  • Correctness needs an external anchor (llama.cpp llama-embedding on the same
    GGUF), not just self-consistency — cosine similarity against a reference
    run, per the project's oracle convention.

Acceptance criteria

  • client.embeddings.create() from the official OpenAI SDK returns a
    well-formed response.
  • Cosine similarity vs a llama.cpp reference on the same model/quant is within
    a stated tolerance, with the tolerance justified.

Activity

  1. jamesburton commented on Sep 21, 2026

    @jamesburton
    OwnerAuthor

    Implemented on branch issue/451-embeddings (5 commits, not pushed).

    Backend coverage is CPU only, deliberately and explicitly. The pooled hidden state comes from a new IEmbeddingModel.ForwardHidden, which only the CPU TransformerModel implements. IEmbeddingModel is a separate interface from IModel precisely so the Vulkan and CUDA models do not silently acquire an unvalidated embeddings path: when one of those is loaded, POST /v1/embeddings returns 501 Not Implemented naming the model type. A GPU path is a follow-on, not a blocker. This is stated in docs/SERVER.md, in the endpoint's XML docs, and on roadmap step 34.

    What landed

    • ForwardHidden stops at llama.cpp's result_norm — post final output norm, pre LM head (src/models/llama.cpp: res->t_embd = cur;), which is the tensor llm_graph_context::build_pooling operates on.
    • EmbeddingPooler: last / mean / cls, semantics taken from build_inp_mean (uniform 1/n) and build_inp_cls (row select); L2 normalisation matching common_embd_normalize step for step.
    • GGUF {arch}.pooling_type is read into ModelConfig.PoolingType using llama.cpp's own enum values. Precedence: explicit request → checkpoint declaration → last. The final fallback is a documented deviation from llama.cpp's NONE default, which is not representable in an OpenAI response.
    • input as string / string[] / int[] / int[][], encoding_format: float|base64, usage.prompt_tokens.

    External anchor. Reference vectors captured from llama.cpp llama-server --embeddings (b9672, -ngl 0) on the same GGUF — llama-embedding is not in the prebuilt release package on this box, but it is the same library and the same build_pooling graph. Committed with full provenance plus the capture script. Cosine vs that reference, SmolLM2-135M-Instruct Q8_0, two inputs:

    pooling input 0 input 1
    last 0.99985868 0.99975375
    mean 0.99946456 0.99993534
    cls 0.99998587 0.99998502

    Gate is cosine ≥ 0.9974, placed at the log-midpoint of the measured gap to the broken forms (worst broken form 0.98717), giving ~4.9× headroom each side. Every broken form — wrong pooling mode, off-by-one row, off-by-one token count, wrong graph point — is asserted to fall below it, and six deliberate mutations were run to confirm the tests actually fail against them. Thread count is ruled out as a source of drift: the same six cosines come out bit-identical single-threaded.

    One thing worth flagging beyond the issue's scope. The server request gate alone did not make this safe. ContinuousBatchSchedulerService's run loop drives forward passes on the same model outside that gate by design, and it is enabled by default for paged-KV CPU models. Removing the fix and running a /v1/completions generation alongside /v1/embeddings aborts the process (CountdownEvent ... below zero out of the shared ComputeThreadPool). Fixed by adding ContinuousBatchSchedulerService.AcquireModelAsync() — the run loop takes the same lock around each Step, so an out-of-band forward interleaves between steps. Cost is one uncontended semaphore per forward pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions