Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
- **Embeddings are available through `mlxcel embed` and the OpenAI-compatible `POST /v1/embeddings` endpoint** (#1408, #1410-#1416). The server can load an embedding checkpoint alone or next to a chat model with `--embedding-model`; supported families cover BERT/XLM-RoBERTa, ModernBERT, SigLIP text, EmbeddingGemma, Qwen3 and Qwen3-VL embeddings, bidirectional Llama/Nemotron/LFM2.5, Llama-Nemotron-VL, and ColBERT-style multimodal late-interaction models. Pooling, L2 normalization, Matryoshka dimensions, token and image inputs, bounded worker admission, and per-checkpoint length limits are shared by the CLI and server paths.
- **Reranking is available through `mlxcel rerank` and the Cohere/Jina-compatible `POST /v1/rerank` endpoint** (#1417). One-label BERT/XLM-RoBERTa/ModernBERT cross-encoders, Qwen3 generative rerankers, and Qwen3-VL multimodal rerankers return ranked relevance probabilities; `--reranker-model` can serve one beside the chat and embedding workers.

### Fixed

- **`mlxcel serve` and `mlxcel-server` match the low-risk llama-server b10621 deployment surface** (#1430). Canonical `LLAMA_ARG_*` variables and `--temperature` now work on both entry points, server sampling defaults match the nightly, OpenAI-shaped requests accept llama aliases, scalar stops and seed `-1`, safe health/reranking route aliases reach the existing handlers, `f16` maps exactly to FP16 split-cache storage, and unsupported cache-reuse chunk sizes or GGML cache quantizers fail instead of changing unrelated behavior.

## [v0.6.0] - 2026-08-22

### Changed
Expand Down
2 changes: 2 additions & 0 deletions docs/embeddings.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ Relevance scoring lives on the same page, under [Reranking](#reranking-v1rerank-
| POST | `/embeddings` | Alias without the `/v1` prefix. |
| POST | `/v1/rerank` | Score a query against a document list. Cohere and Jina compatible; see [Reranking](#reranking-v1rerank-and-mlxcel-rerank). |
| POST | `/rerank` | Alias without the `/v1` prefix. |
| POST | `/v1/reranking` | llama-server alias for `/v1/rerank`. |
| POST | `/reranking` | llama-server alias for `/rerank`. |
| GET | `/v1/models` | Lists the embedding model, and the reranker, next to the chat model when `--embedding-model` / `--reranker-model` load a second checkpoint. |

## Implementation source map
Expand Down
23 changes: 20 additions & 3 deletions docs/environment-variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,25 @@ and cached on first use. Set them before starting `mlxcel` or `mlxcel-server`.
They exist for benchmarking, rollback, or kernel-development work and may
change between releases.

### llama-server compatibility variables

Both server entry points accept the canonical llama-server b10621 variables below. An explicit CLI flag wins. Where a legacy mlxcel spelling is listed, the canonical variable wins over that legacy alias.

| Canonical variable | Flag | Retained legacy alias |
|--------------------|------|-----------------------|
| `LLAMA_ARG_BATCH` | `--batch-size` | `LLAMA_ARG_BATCH_SIZE` |
| `LLAMA_ARG_UBATCH` | `--ubatch-size` | `LLAMA_ARG_UBATCH_SIZE` |
| `LLAMA_ARG_SPEC_DRAFT_MODEL` | `--draft-model` / `--model-draft` | `LLAMA_ARG_MODEL_DRAFT` |
| `LLAMA_ARG_THINK_BUDGET` | `--reasoning-budget` | `LLAMA_ARG_REASONING_BUDGET` |
| `LLAMA_ARG_LOG_FILE` | `--log-file` | `LLAMA_LOG_FILE` |
| `LLAMA_ARG_CHAT_TEMPLATE` | `--chat-template` | — |
| `LLAMA_ARG_CHAT_TEMPLATE_FILE` | `--chat-template-file` | — |
| `LLAMA_ARG_ENDPOINT_METRICS` | `--metrics` | — |
| `LLAMA_ARG_ENDPOINT_PROPS` | `--props` | — |
| `LLAMA_ARG_ENDPOINT_SLOTS` | `--slots` | — |

`LLAMA_ARG_CACHE_REUSE` is an integer minimum reuse chunk size in llama-server, not a prompt-cache enable switch. mlxcel accepts `0` without changing prompt-cache enablement and rejects positive values with an unsupported-setting error. Use `MLXCEL_PROMPT_CACHE_ENABLED` to enable or disable the cache.

## Common runtime variables

| Variable | Values | Default | Notes |
Expand Down Expand Up @@ -137,9 +156,7 @@ These variables are applied when the corresponding CLI flag is absent.
| `APC_NUM_BLOCKS` | unsigned integer | derived from max entries | `--apc-num-blocks` |
| `APC_HASH` | `sha256` or `blake3` | `sha256` | `--apc-hash` |

`MLXCEL_PROMPT_CACHE_ENABLED` has higher precedence than the llama.cpp
compatibility alias `LLAMA_ARG_CACHE_REUSE` when both are set and no CLI flag is
provided.
`LLAMA_ARG_CACHE_REUSE` is validated independently of `MLXCEL_PROMPT_CACHE_ENABLED`; it is not a boolean alias for this table's enable switch.

Automatic Prefix Caching is on by default; pass `--apc-enabled=false` or set
`APC_ENABLED=false` to fall back to whole-prefix matching only (a stored prefix
Expand Down
4 changes: 3 additions & 1 deletion docs/turbo-kv-cache.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ Supported split-flag combinations:

| `--cache-type-k` | `--cache-type-v` | Result |
|------------------|------------------|--------|
| `fp16` | `fp16` | `fp16` |
| `fp16` or `f16` | `fp16` or `f16` | `fp16` |
| `int8` | `int8` | `int8` |
| `fp16` | `turbo4` or `turbo4-asym` | `fp16+turbo4` |
| `turbo4` | `turbo4` | `turbo4` |
Expand All @@ -124,6 +124,8 @@ Supported split-flag combinations:
If split flags and `--kv-cache-mode` are both supplied, split flags take
precedence and a warning is logged.

The exact llama.cpp `f16` spelling aliases FP16 storage. GGML quantizer names such as `q8_0` and `q4_0` remain errors because mlxcel has no mathematically equivalent cache format for them.

Environment variables:

| Variable | Purpose |
Expand Down
Loading