Skip to content

Add exact MoVA serving to xLLM - #47

Draft
akhauriyash wants to merge 3 commits into
prodfrom
yash.akhauri/mova-sglang
Draft

Add exact MoVA serving to xLLM#47
akhauriyash wants to merge 3 commits into
prodfrom
yash.akhauri/mova-sglang

Conversation

@akhauriyash

Copy link
Copy Markdown
Collaborator

What changed

  • Adds native MoVA attention for the xLLM model family.
  • Packs Q/K/gate projections and output-shards routed value experts for TP serving.
  • Reuses ordinary paged K/routed-V caching, including prefix caching and chunked prefill.
  • Preserves selection-only FP32 router-bias semantics, grouped RMSNorm, interleaved RoPE, and the softplus attention gate.
  • Adds fail-fast validation for unsupported phase-one configurations.

Why

MoVA RL rollouts need a native SGLang implementation with CUDA-graph-safe kernels and a stable weight-loading contract. The existing xLLM vLLM bridge is a useful correctness reference but is not the production SGLang path.

Validation

  • 25 focused tests covering router/value math, TP sharding/loading, mappings, RoPE, norms, CUDA graph replay, and fullgraph compile.
  • Real 36B TP8 BF16 acceptance: load, cached decode, concurrent batching, radix-prefix reuse, and chunked prefill.
  • A 16-prompt HF-vs-SGLang parity corpus and corrected real-model production rerun are in progress; keep this draft until those gates are recorded.

Serve the 36B MoVA checkpoint with packed Q/K/gate projections, output-sharded routed values, native router and norm semantics, ordinary RadixAttention caching, and live-update mappings. Reuse the fused-MoE first GEMM behind a compile-safe custom-op boundary and preserve the legacy K2 path.
Transformers 5 materializes a null rope_scaling field as an explicit default-RoPE dictionary. Treat that normalized representation as unscaled while retaining the fail-fast guard for actual scaling modes.
xLLM applies sigmoid, correction bias, expert selection, and top-k normalization in FP32 after the BF16 router projection. Match that contract in the native fallback so near-boundary routes do not change during serving.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant