feat(model-serving): the muse-glimmer-30b preset on vLLM's own image

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **`muse-glimmer-30b`: Meta's Muse Glimmer 30B on one 48 GB GPU, on vLLM's own server image** (giantswarm/agent-platform#597). Red Hat AI's FP8 block quantization (`RedHatAI/Muse-Glimmer-30B-FP8-block`, 32 GiB, served from the signed model image `giantswarm/models/muse-glimmer-30b-fp8:1deb4641ff84`) at 32k context, four concurrent requests, one image per prompt, reasoning and tools with vLLM's `muse_glimmer` parsers under vLLM v0.30.0's Muse Glimmer chat template (`chat-templates/muse-glimmer-30b.jinja`). The model's architecture and parsers first ship in vLLM 0.28.0 and the llm-d runtime carries an older vLLM, so the preset names its runtime in `spec.template`: `gsoci.azurecr.io/giantswarm/vllm-openai:v0.30.0`, the signed copy of `vllm/vllm-openai` that retagger pins by digest (giantswarm/retagger#1245). The well-known template's command runs it unchanged. Sized for a `g6e.2xlarge` or `g6e.4xlarge`.
 - **`clusterManager.networkPolicy.workloadClusters.provider: aws` opens the CAPA API server load balancers, `*.*.elb.amazonaws.com`** (giantswarm/agent-platform#487). cluster-manager dials the `server:` of a cluster's Cluster API kubeconfig Secret, which on CAPA is the API server's load balancer, never the `api.*.<base domain>` pattern the values used to document. The preset renders next to `fqdns` in cluster-manager's egress and in model-manager's, which follows cluster-manager's block. An unknown provider is refused and the presets are named. The values now say which host to select and that Cilium's `*` matches one DNS label.
 - **The Klaus Gateway board shows the whole turn as a person sees it: "Turn done (final flush), p50 and p95"** (giantswarm/giantswarm#37787). The panel sits beside "Time to first text" and shows the time from the message's arrival to the last edit of the answer in the channel, from `klaus_gateway_turn_phase_seconds{phase="final_flush"}`. Before, the board showed only this phase's p95, inside "Turn phases, p95".
 

Makefile.custom.mk

@@ -2602,7 +2602,7 @@ verify-wiring: ## Assert the standalone's ported wiring: toggles off = no object
 		grep -q -e "$$pattern" /tmp/vw-ms.out || { echo "FAIL: the model serving render lacks $$pattern"; exit 1; }; \
 	done
 	@if grep -qE 'serving\.kserve\.io|ClusterServingRuntime|kind: InferenceService|kserve-controller-manager|spec\.runtime|^      runtimes?:' /tmp/vw-ms.out; then echo "FAIL: the serving render carries a classic serving object or key"; grep -nE 'serving\.kserve\.io|ClusterServingRuntime|InferenceService|kserve-controller-manager|runtimes?:' /tmp/vw-ms.out | head; exit 1; fi
-	@[ "$$(grep -c 'agent-platform.giantswarm.io/serving-preset: "true"' /tmp/vw-ms.out)" = "13" ] || { echo "FAIL: expected the 13 shipped presets, got $$(grep -c 'agent-platform.giantswarm.io/serving-preset: "true"' /tmp/vw-ms.out)"; exit 1; }
+	@[ "$$(grep -c 'agent-platform.giantswarm.io/serving-preset: "true"' /tmp/vw-ms.out)" = "14" ] || { echo "FAIL: expected the 14 shipped presets, got $$(grep -c 'agent-platform.giantswarm.io/serving-preset: "true"' /tmp/vw-ms.out)"; exit 1; }
 	@if grep -q 'kind: NetworkPolicy' /tmp/vw-ms.out; then echo "FAIL: a kubernetes NetworkPolicy rendered under the cilium flavor"; exit 1; fi
 	@if grep -q '^kind: PersistentVolumeClaim' /tmp/vw-ms.out; then echo "FAIL: the cache claim rendered as a release resource (Helm's wait would wait for a Bind only the first predictor brings: #483)"; exit 1; fi
 	@echo "ok: model serving fleet shape"

README.md

@@ -406,7 +406,7 @@ https://models.<cluster>.<base domain>/<namespace>/<model>/v1/chat/completions
 
 **The presets.** Four 24 GB recipes for one L4-class GPU, the September 2026 line-up (giantswarm/agent-platform#591), each served from a signed model image under `gsoci.azurecr.io/giantswarm/models/` ([Models as OCI images](#the-serving-slice-and-the-models-gateway)) with tools and reasoning on, 32k context, four concurrent requests, `weightsGiB + overheadGiB <= 24`: `gpt-oss-20b` (OpenAI's 21B mixture of experts at its native MXFP4, 13 GiB; `--tool-call-parser=openai --reasoning-parser=openai_gptoss`; a card without native MXFP4 runs the experts through vLLM's Marlin kernel, which its backend selection picks on its own; the harmony renderer reads its tiktoken encodings from the model image, `TIKTOKEN_ENCODINGS_BASE=/mnt/models/tiktoken`, since the pod has no egress to fetch them), `gemma-4-12b` (Google's official QAT W4A16 checkpoint, 10 GiB, one image per request, the `gemma4` tool and reasoning parsers, thinking off unless a request asks), `qwen3-5-9b-fp8` (Red Hat's FP8 dynamic quantization of Qwen3.5 9B, 14 GiB with the MTP head, `qwen3` reasoning and `qwen3_coder` tool parsers, thinking off by default, one image per request) and `qwen3-5-4b` (Qwen3.5 4B in BF16, 9 GiB, the same parsers and the most KV cache of the four). All four are sized for the smallest L4 instance, a `g6.xlarge` (4 vCPU, 16 GiB): requests 2 vCPU / 10 GiB, limits 4 vCPU / 12 GiB — a Giant Swarm node of that shape reports 3.4 vCPU / 13.4 GiB allocatable and the daemonsets that follow the pool's taint take ~0.4 vCPU / 1.5 GiB more, so a preset requesting 4 vCPU / 16 GiB sat Pending for ever on a pool of `xlarge` nodes while Karpenter refused the size (giantswarm/agent-platform#502). A preset an installation adds under `modelServing.presets` for the same pool should stay within 3 vCPU / 11.9 GiB; cluster-manager's `create_node_pool` answers with what each of a pool's sizes leaves a predictor and which size hosts each published preset. None names a serving image: the predictor runs the `llm-d-cuda` the well-known config names; the preset schema's new optional `spec.template` carries `LLMInferenceService` template fields verbatim, an image there only with a stated reason. Every shipped preset's `weightsGiB` is what the Hub holds: `make verify-preset-weights` (in `verify-all`, so on every PR) sizes each preset's `spec.model.id` the way model-manager's `check_fit` does — the safetensors index's `total_size`, else the safetensors files — and fails a preset below the Hub's size, more than 15 % above it, or one the Hub cannot size, so cluster-manager's `presetFit` and model-manager's `check_fit` judge a pool from the same number (giantswarm/agent-platform#535).
 
-**The 48 GB presets.** Three recipes for one 48 GB GPU of the L40S family (`g6e.2xlarge` or `g6e.4xlarge`; requests 4 vCPU / 32 GiB, limits 8 / 48, `--gpu-memory-utilization=0.92`, four concurrent requests, tools on, one image per prompt), each served from a signed model image (giantswarm/agent-platform#591). `qwen3-8-27b-l40s` serves Qwen3.8 27B as NVIDIA's NVFP4 checkpoint (20.4 GiB: NVFP4 on the MLPs and the language-model head, FP8 on the attention layers; the GPU has no native FP4, so vLLM takes the Marlin weight-only path for the NVFP4 layers; 64k context, thinking off by default, no speculative decoding) — the shipped `qwen3-8-27b` is the same model tuned for a 128 GB unified-memory Blackwell node (DFlash2 speculative decoding, 256k context, a patched chat template) and hosts on no curated AWS size (giantswarm/agent-platform#544). `qwen3-6-35b-a3b` serves Qwen3.6 35B-A3B, Qwen's FP8 mixture of experts with 3B active per token (35 GiB, 32k context, the `qwen3` reasoning and `qwen3_coder` tool parsers, thinking off by default). `gemma-4-31b` serves Gemma 4 31B as Red Hat's FP8-dynamic quantization (31 GiB, 64k context, the `gemma4` tool and reasoning parsers under vLLM's Gemma 4 chat template, shipped as `chat-templates/gemma-4-31b.jinja`; audio off). Meta's Muse Glimmer 30B, the line-up's fourth 48 GB model, ships no preset yet: vLLM serves its architecture and its `muse_glimmer` parsers from v0.28.0, and the platform's runtime image (llm-d v0.8.0) carries vLLM v0.23.0 — the preset joins once the well-known runtime image carries vLLM 0.28 or later. A model whose weights exceed the card it targets ships no preset: `make verify-preset-weights` sizes every shipped preset's `weightsGiB` against the Hub, and `weightsGiB + overheadGiB` stays within the card.
+**The 48 GB presets.** Four recipes for one 48 GB GPU of the L40S family (`g6e.2xlarge` or `g6e.4xlarge`; requests 4 vCPU / 32 GiB, limits 8 / 48, `--gpu-memory-utilization=0.92`, four concurrent requests, tools on, one image per prompt), each served from a signed model image (giantswarm/agent-platform#591). `qwen3-8-27b-l40s` serves Qwen3.8 27B as NVIDIA's NVFP4 checkpoint (20.4 GiB: NVFP4 on the MLPs and the language-model head, FP8 on the attention layers; the GPU has no native FP4, so vLLM takes the Marlin weight-only path for the NVFP4 layers; 64k context, thinking off by default, no speculative decoding) — the shipped `qwen3-8-27b` is the same model tuned for a 128 GB unified-memory Blackwell node (DFlash2 speculative decoding, 256k context, a patched chat template) and hosts on no curated AWS size (giantswarm/agent-platform#544). `qwen3-6-35b-a3b` serves Qwen3.6 35B-A3B, Qwen's FP8 mixture of experts with 3B active per token (35 GiB, 32k context, the `qwen3` reasoning and `qwen3_coder` tool parsers, thinking off by default). `gemma-4-31b` serves Gemma 4 31B as Red Hat's FP8-dynamic quantization (31 GiB, 64k context, the `gemma4` tool and reasoning parsers under vLLM's Gemma 4 chat template, shipped as `chat-templates/gemma-4-31b.jinja`; audio off). Meta's Muse Glimmer 30B, the line-up's fourth 48 GB model, ships no preset yet: vLLM serves its architecture and its `muse_glimmer` parsers from v0.28.0, and the platform's runtime image (llm-d v0.8.0) carries vLLM v0.23.0 — the preset joins once the well-known runtime image carries vLLM 0.28 or later. A model whose weights exceed the card it targets ships no preset: `make verify-preset-weights` sizes every shipped preset's `weightsGiB` against the Hub, and `weightsGiB + overheadGiB` stays within the card. `muse-glimmer-30b` serves Meta's Muse Glimmer 30B as Red Hat AI's FP8 block quantization (32 GiB, 32k context, vision, reasoning, tools with vLLM's `muse_glimmer` parsers under vLLM v0.30.0's Muse Glimmer chat template) on vLLM's own server image, `gsoci.azurecr.io/giantswarm/vllm-openai:v0.30.0` (the signed copy retagger pins by digest): the model's architecture and parsers first ship in vLLM 0.28.0, newer than the llm-d runtime's vLLM, so this is the one shipped preset that names its runtime image, until an llm-d release carries vLLM 0.28 or later (giantswarm/agent-platform#597).
 
 **The Qwen3.8-Flash-Next preset.** `qwen3-8-flash-next-nvfp4` serves Qwen3.8-Flash-Next (a 125B mixture of experts with 6B active, a 51B n-gram embedding table, 262k context, tools, reasoning, vision) on one 128 GB unified-memory Blackwell node from an OCI model image (giantswarm/agent-platform#553). NVIDIA's NVFP4 checkpoint is 123.6 GiB on disk and does not fit the node next to a KV cache, so the preset serves the quantization-aware-distilled repack `local-inference-lab/Qwen3.8-Flash-Next-NVFP4` (98.6 GiB, the n-gram table in NVFP4 too) with the table read from disk (`VLLM_PLE_TABLE_MEMORY=disk`) and MTP speculative decoding (four draft tokens): `weightsGiB: 99` (the Hub's size), `overheadGiB: 20` (about 74 GiB resident; `--gpu-memory-utilization=0.80` of the 128 GB leaves the rest to the fp8 KV cache), eight concurrent requests at 4096 batched tokens, the `qwen3` reasoning parser, tools on with the `qwen3_xml` parser. It is the first shipped preset served from a model image. Its serving image is the B12X serving stack — the upstream vLLM image cannot serve this architecture on the node — which the installation names as the runtime image of the well-known `LLMInferenceServiceConfig` such a node serves with (`kserve-runtime-configs`, [The well-known configs](#the-serving-slice-and-the-models-gateway)); the preset itself names no image. `spec.model.storageUri` is `oci://gsoci.azurecr.io/giantswarm/models/qwen3-8-flash-next-nvfp4:7c4f1bc1a2d6` (the tag is the first 12 characters of the checkpoint revision); `modelServing.modelImages.registry` swaps the host for an installation's own registry that holds the same path, and `modelServing.prepull.modelPresets` pre-pulls the image onto the serving nodes. `HOME`, `HF_HOME` and `VLLM_CACHE_ROOT` sit under `/tmp` because the modelcar path runs the runtime as a non-root uid on a read-only image. `make verify-wiring` expects thirteen shipped presets; `verify-preset-weights` and the eval check of `verify-serving-slice` cover the file; `verify-gpu-pool` leaves the preset's ConfigMap and its discovery entry out of the golden comparison while `origin/main` predates it.
 

helm/agent-platform-connectivity/files/model-serving/chat-templates/muse-glimmer-30b.jinja

@@ -0,0 +1 @@
+{%- macro render_content(content) -%}{%- if content is string -%}{{- content -}}{%- elif content is not none -%}{%- for part in content -%}{%- if part['type'] == 'image' -%}{{- '<|image|>' -}}{%- elif part['type'] == 'video' -%}{{- '<|video|>' -}}{%- elif part['type'] == 'text' -%}{{- part['text'] -}}{%- endif -%}{%- endfor -%}{%- endif -%}{%- endmacro -%}{%- macro render_atem(tc) -%}{%- set args = tc.function.arguments -%}{%- if args is not mapping -%}{{- raise_exception('MuseGlimmer ATEM chat template requires tool_call.function.arguments to be a dict (mapping); a JSON string cannot be parsed in the HF jinja sandbox.') -}}{%- endif -%}{{- '<atem:function_calls>\n<atem:invoke name="' + tc.function.name + '">\n' -}}{%- for k, v in args.items() -%}{{- '<atem:parameter name="' + k + '">' -}}{%- if v is boolean -%}{%- if v -%}true{%- else -%}false{%- endif -%}{%- elif v is none -%}null{%- elif v is mapping or (v is iterable and v is not string) -%}{{- v | tojson -}}{%- else -%}{{- v -}}{%- endif -%}{{- '</atem:parameter>\n' -}}{%- endfor -%}{{- '</atem:invoke>\n</atem:function_calls>' -}}{%- endmacro -%}{%- macro render_tool_defs(tools) -%}{{- 'In this environment you have access to a set of tools you can use to answer the user\'s question.\n\n' -}}{{- 'You can invoke a function by writing a "<atem:function_calls>" block like the following:\n' -}}{{- '<atem:function_calls>\n<atem:invoke name="$FUNCTION_NAME">\n<atem:parameter name="$PARAMETER_NAME">$PARAMETER_VALUE</atem:parameter>\n...\n</atem:invoke>\n</atem:function_calls>\n\n' -}}{{- 'String and scalar parameters should be specified as is, while lists and objects should use JSON format. Note that spaces for string values are not stripped. The output is not expected to be valid XML and is parsed with regular expressions.\n' -}}{{- 'Here are the functions available in JSONSchema format:\n' -}}{{- '// Tool metadata\n' -}}{%- set nsns = namespace(seen=[]) -%}{%- for tool in tools -%}{%- set fn = tool.function if tool.function is defined else tool -%}{%- set tns = fn.name.split('.')[0] -%}{%- if tns not in nsns.seen -%}{%- set nsns.seen = nsns.seen + [tns] -%}{%- endif -%}{%- endfor -%}{%- set nd = tool_namespace_descriptions if tool_namespace_descriptions is defined else {} -%}{%- for tns in nsns.seen -%}{{- '{"name": ' + (tns | tojson) + ', "description": ' + ((nd[tns] if tns in nd else '') | tojson) + '}\n' -}}{%- endfor -%}{{- '// Function schemas' -}}{%- for tool in tools -%}{%- set fn = tool.function if tool.function is defined else tool -%}{{- '\n{"name": ' + (fn.name | tojson) + ', "description": ' + (fn.description | tojson) + ', "parameters": ' + (fn.parameters | tojson) + '}' -}}{%- endfor -%}{{- '\n\nHere\'s an example of how to call a function in the tool set. To make parallel tool calls, emit each call as its own message (one tool call per message) in consecutive messages:\n' -}}{{- '(If the tool namespace is not specified, invoke the function directly as `example_function_name` rather than `example_tool_name.example_function_name`)\n\n' -}}{{- 'to=example_tool_name.example_function_name\n\n' -}}{{- '<atem:function_calls>\n<atem:invoke name="example_tool_name.example_function_name">\n' -}}{{- '<atem:parameter name="example_parameter_1">value_1</atem:parameter>\n' -}}{{- '<atem:parameter name="example_parameter_2">This is the value for the second parameter\nthat can span\n"multiple" lines\n</atem:parameter>\n' -}}{{- '</atem:invoke>\n</atem:function_calls>' -}}{%- endmacro -%}{%- macro render_reasoning() -%}{%- set rs = reasoning_strength if reasoning_strength is defined and reasoning_strength else 'high' -%}{{- 'Reasoning strength: ' + rs + '.' -}}{%- endmacro -%}{%- macro render_system_meta(tools) -%}{%- set rns = namespace(recipients=['"self"'], nslist=[]) -%}{%- if tools -%}{%- for tool in tools -%}{%- set fn = tool.function if tool.function is defined else tool -%}{%- set tns = fn.name.split('.')[0] -%}{%- if tns not in rns.nslist -%}{%- set rns.nslist = rns.nslist + [tns] -%}{%- endif -%}{%- endfor -%}{%- for tns in rns.nslist -%}{%- set rns.recipients = rns.recipients + ['"' + tns + '.*"'] -%}{%- endfor -%}{%- endif -%}{%- set rns.recipients = rns.recipients + ['"user"'] -%}{{- '# Valid recipients: ' + rns.recipients | join(', ') + '.' -}}{%- endmacro -%}{{- bos_token -}}{%- set ns = namespace(has_system=false) -%}{%- for m in messages -%}{%- if m['role'] == 'system' -%}{%- set ns.has_system = true -%}{%- endif -%}{%- endfor -%}{%- if not ns.has_system and (add_generation_prompt or tools) -%}{{- '<|start|>system<|message|>You are a helpful AI assistant.' -}}{%- set kc = knowledge_cutoff if knowledge_cutoff is defined and knowledge_cutoff else '2026-01-04' -%}{{- '\nKnowledge cutoff: ' + kc + '.' -}}{%- if current_date is defined and current_date -%}{{- '\nCurrent date: ' + current_date + '.' -}}{%- elif strftime_now is defined -%}{{- '\nCurrent date: ' + strftime_now('%Y-%m-%d') + '.' -}}{%- endif -%}{{- '\n\n' -}}{{- render_reasoning() -}}{%- if tools -%}{{- '\n\n' -}}{{- render_tool_defs(tools) -}}{%- endif -%}{{- '\n\n' -}}{{- render_system_meta(tools) -}}{{- '<|eot|>' -}}{%- endif -%}{%- for message in messages -%}{%- set role = message['role'] -%}{%- set end_token = '<|eom|>' if (not loop.last and messages[loop.index0 + 1]['role'] == role) else '<|eot|>' -%}{%- if role == 'system' -%}{{- '<|start|>system<|message|>' -}}{{- render_content(message['content']) -}}{{- '\n\n' -}}{{- render_reasoning() -}}{%- if tools -%}{{- '\n\n' -}}{{- render_tool_defs(tools) -}}{%- endif -%}{{- '\n\n' -}}{{- render_system_meta(tools) -}}{{- '<|eot|>' -}}{%- elif role == 'user' -%}{{- '<|start|>user<|message|>' -}}{{- render_content(message['content']) -}}{{- '<|eot|>' -}}{%- elif role == 'tool' -%}{%- set tname = message.get('name') -%}{%- if not tname -%}{%- set tcid = message.get('tool_call_id') -%}{%- set rns = namespace(name=tcid if tcid else '') -%}{%- for m in messages -%}{%- if m.get('tool_calls') -%}{%- for tc in m['tool_calls'] -%}{%- if tcid is not none and tc.id is defined and tc.id == tcid -%}{%- set rns.name = tc.function.name -%}{%- endif -%}{%- endfor -%}{%- endif -%}{%- endfor -%}{%- set tname = rns.name -%}{%- endif -%}{{- '<|start|>tool ' + tname + '<|message|><tool_output name="' + tname + '">\n' -}}{{- render_content(message['content']) -}}{{- '\n</tool_output><|eot|>' -}}{%- elif role == 'assistant' -%}{%- if message.get('reasoning_content') -%}{{- '<|start|>assistant to=self<|message|>' + message['reasoning_content'] + '<|eom|>' -}}{%- endif -%}{%- if message.get('tool_calls') -%}{%- for tc in message['tool_calls'] -%}{{- '<|start|>assistant to=' + tc.function.name + '<|message|>' -}}{{- render_atem(tc) -}}{%- if loop.last -%}{{- end_token -}}{%- else -%}{{- '<|eom|>' -}}{%- endif -%}{%- endfor -%}{%- else -%}{%- set recipient = message.get('recipient') or 'user' -%}{%- set end_turn = message.get('end_turn') -%}{%- if end_turn is none -%}{%- set end_turn = not (recipient and recipient != 'user') -%}{%- endif -%}{{- '<|start|>assistant' -}}{%- if recipient -%}{{- ' to=' + recipient -}}{%- endif -%}{{- '<|message|>' -}}{{- render_content(message['content']) -}}{{- ('<|eot|>' if end_turn else '<|eom|>') -}}{%- endif -%}{%- endif -%}{%- endfor -%}{%- if add_generation_prompt -%}{{- '<|start|>assistant' -}}{%- endif -%}