kserve: the fit lets FP8 weights land on a GPU without native FP8, where vLLM answers wrong #264

Problem

The shipped preset qwen3-5-9b-fp8 (RedHatAI/Qwen3.5-9B-FP8-dynamic, compressed-tensors FP8 weights and activations) served through model-manager on a g5.xlarge pool (A10G: Ampere, compute capability 8.6) of a test installation goes Ready and answers gibberish, deterministically at temperature 0 ("What is the capital of France? Answer in one word." → -害愁一气龙林丨驾国 暗胞一刚豫们更胜法语 的的闪 溜immerfeld…). vLLM logs Your GPU does not have native support for FP8 computation but FP8 quantization is being used. Weight-only FP8 compression will be used leveraging the Marlin kernel. The same preset answers correctly on an L4 (Ada, 8.9); qwen3-5-4b (BF16) answers correctly on the same A10G nodes. The runtime image is not the cause: llm-d-fast/llm-d-cuda:v0.8.0 and llm-d-slim/llm-d-cuda:v0.8.0 garble alike, as do --enforce-eager and --dtype=float16. check_fit and load_model said the preset fits the g5.xlarge (14 + 9 GiB within 24 GiB), so a person gets a Ready model that is wrong.

Root cause

The fit judges a GPU by its memory and the KV cache and never by its generation. The checkpoint's config.json, which the fit already reads for the KV cache layout, says the weights are FP8 (quantization_config), and vLLM runs FP8 natively only from compute capability 8.9 (Ada, Hopper, Blackwell); below it the weight-only FP8 Marlin fallback answers wrong for this checkpoint. A pool's instance family (g5: A10G) and a node's nvidia.com/gpu.compute.major/minor labels name the generation; neither is consulted.

Proposed solution

The fit judges the GPU generation beside the memory and the KV cache: from config.json the weights' need (FP8 — quant_method: fp8, compressed-tensors 8-bit float, ModelOpt FP8 — needs 8.9; NVFP4 needs 10.0; bf16 needs 8.0) against a node's compute capability labels or, for a pool with no node yet, the GPU of its instance family (g4dn T4 7.5, g5 A10G 8.6, g6 L4 8.9, g6e L40S 8.9, …). A GPU below the need is a refusal whose reason names both and the way out (a pool of 8.9 or above, or a BF16 checkpoint); load_model composes nothing. Among several pools, one whose GPU is below the need is skipped for one that is not. The answer carries computeCapabilityRequired and computeCapability.

Acceptance criteria

• check_fit for qwen3-5-9b-fp8 against a g5.xlarge pool with no node yet answers fits: false, naming compute capability 8.9 against the A10G's 8.6; against a g6.xlarge it fits as before.
• On a node labelled nvidia.com/gpu.compute.major=8, nvidia.com/gpu.compute.minor=6 the preset is refused; a bf16 preset is not.
• load_model of the preset on an installation whose only GPU pool is A10G creates no LLMInferenceService and returns the refusal.
• Unit tests cover the shipped checkpoints' quantization configs (compressed-tensors FP8, vLLM fp8, ModelOpt, int4, MXFP4, bf16).

Beyond this repository: cluster-manager's form-time fit (create_node_pool's presetFit) still reports the FP8 preset as hosted by a g5 size (giantswarm/cluster-manager#178), and the preset contract declares no GPU-generation floor the form could read (giantswarm/agent-platform#832). Fix: #265.