Skip to content

model-serving: a Muse Glimmer 30B preset once the runtime carries vLLM 0.28 or later #597

Description

@teemow

Problem

The signed model image for Meta's Muse Glimmer 30B (RedHatAI FP8-block repack, Apache 2.0) exists, but the platform's llm-d runtime image (llm-d-cuda v0.8.0) carries vLLM 0.23.0, and the model's architecture and its muse_glimmer tool and reasoning parsers first appear in vLLM v0.28.0 (vllm-project/vllm#51655). No llm-d release carries 0.28.0 yet (v0.9.0 = vLLM 0.26.0). A preset today would never start.

Proposed solution

  • When an llm-d release with vLLM ≥ 0.28.0 exists and its image is mirrored and signed on gsoci: bump the runtime in kserve-runtime-configs, then add the preset muse-glimmer-30b on oci://gsoci.azurecr.io/giantswarm/models/muse-glimmer-30b-fp8:1deb4641ff84 (32 GiB FP8, one 48 GB GPU, contextLength: 32768, muse_glimmer parsers, image 1).
  • Alternatively, a per-preset runtime image override on a signed mirror of a vLLM ≥ 0.28 image, if that mechanism ships first.

Acceptance criteria

  • The runtime serves MuseGlimmerForConditionalGeneration (a serve in the lab or on a test installation reaches Ready and answers a tool call).
  • The preset ships and verify-preset-weights passes.

Follows #591.

Activity

  1. teemow commented on Sep 26, 2026

    @teemow
    MemberAuthor

    From Timo's agent: proven on gazelle, 2026-09-26, one L40S (g6e.4xlarge). The object is #673's preset rendered as an LLMInferenceService: image gsoci.azurecr.io/giantswarm/vllm-openai:v0.30.0, model image giantswarm/models/muse-glimmer-30b-fp8:1deb4641ff84, the preset's arguments and its muse_glimmer chat template.

    • vLLM 0.30.0 resolved MuseGlimmerForConditionalGeneration, with a 321,920-token KV cache (9.8× concurrency at 32k).
    • Ready at 07:40Z after a node replacement.
    • A tool-calling request answered HTTP 200 in 3.5 s: finish_reason: tool_calls, get_weather({"city": "Berlin"}), reasoning content from the muse_glimmer parser.

    The first criterion is met. The preset ships with #673.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions