[AMD][AgentX] Kimi-K3 FP4 MI355X agentic TP8xPP2 DSpark / [AMD][AgentX] MI355X 上 Kimi-K3 FP4 agentic TP8×PP2 DSpark

[charxwu]

Summary

• Add kimik3-fp4-mi355x-vllm-agentic-tp8pp2 to amd-master.yaml: Kimi-K3 MXFP4 on MI355X, aggregated TP8×PP2 (2 nodes / 16 GPUs, decode.num-worker: 0 so GPUs are counted once), agentic-coding, conc-list [4, 8, 10, 12, 16, 20], DSpark speculative decoding (spec-decoding: mtp, num-spec-tokens: 2, synthetic acceptance length 2.51).
• Wire the CI path: runners/launch_mi355x-amds.sh → benchmarks/multi_node/agentic/kimik3_fp4_mi355x_vllm.sh → kimik3_agg_pp_job.slurm → kimik3_fp4_mi355x_vllm_tp8pp2.sh (rank-0 OpenAI frontend + agentic replay; rank-1 --headless PP worker).
• Fix CI bring-up bugs found during e2e:
  • Override the vllm-openai-rocm image ENTRYPOINT=vllm with --entrypoint bash (otherwise the serve script path was parsed as a vLLM CLI arg).
  • Resolve model weights across known host roots and verify on all allocated nodes before launch (/model is :ro; missing weights previously mounted an empty dir and failed on hf download with Read-only file system).
• Enable DSpark on this recipe (previously checked in as spec-decoding: none / STP-only) after fixing the CONC=20 stability issue below.
• Add runtime patches under experimental/kimik3-v4/aiter/:
  • patch_kv_offload_block_cap.py — clamp SimpleCPUOffloadConnector CPU block pool to the smallest PP-stage capacity (KV_OFFLOAD_MAX_CPU_BLOCKS); fixes CONC=20 SIGSEGV in hipMemcpyBatchAsync.
  • patch_kv_offload_batch_chunk.py — block-ID range diagnostics (off by default: KV_OFFLOAD_BATCH_CHUNK_PATCH=0).
  • aiter sitecustomize / GEMM merge / n6288 + ca_flush_sync patch stack used by the smoke-validated serve path.
• Raise MAX_NUM_BATCHED_TOKENS 2048 → 8192 for this prefill-heavy agentic workload (mean ISL ~135K, p95 ~455K); dominant throughput lever.
• Append perf-changelog.yaml entries for the e2e wiring and the DSpark flip.

Image: vllm/vllm-openai-rocm:nightly-cb8104839c141609d99f1254459ef3a4f1bd4263
Model: moonshotai/Kimi-K3
KV offload: SimpleCPUOffloadConnector (kv-offloading: dram, kv-offload-backend: vllm-simple, dram-utilization: 0.50 → ~1499 GB/node)
Topology: aggregated TP8 × PP2, ASYNC_SCHEDULING=false (toggleable via env)

Root cause (CONC=20 SIGSEGV)

SimpleCPUOffloadScheduler sizes the shared CPU block-ID space from the global GPU KV-cache config (~47,095 blocks). Each PP stage's connector worker independently sizes its pinned buffer from its own unique-KV-tensor count (PP0: 3 → larger; PP1: 5 → ~28,257 blocks). Once the scheduler issued IDs past PP1's buffer, hipMemcpyBatchAsync wrote OOB → SIGSEGV on all 8 PP1 workers. The vendored block-cap patch clamps to the min stage capacity; upstream fix: vllm-project/vllm#52921 (draft).

Smoke validation (g06+g17, 30-min profile, zero GPU faults)

Conc	Throughput/GPU	TTFT p50	E2E p50	GPU KV occupancy
16	2,049 tok/s	—	—	—
20	2,078 tok/s	1,682 ms	8,963 ms	90.4%

Test plan

• Label full-sweep-fail-fast; green full sweep (incl. evals) on a commit in this PR
• Confirm c4/c8/c10/c12/c16/c20 green with DSpark + artifact upload
• /reuse-sweep-run after full green sweep with evals

Related

• Upstream: vllm-project/vllm#52921
• PR: [AMD][AgentX] Kimi-K3 FP4 MI355X agentic TP8xPP2 DSpark / [AMD][AgentX] MI355X 上 Kimi-K3 FP4 agentic TP8×PP2 DSpark #2678

---

中文说明

• 在 amd-master.yaml 新增 kimik3-fp4-mi355x-vllm-agentic-tp8pp2：MI355X 上 Kimi-K3 MXFP4 的聚合 TP8×PP2（2 节点 / 16 卡，decode.num-worker: 0，避免 GPU 重复计数），agentic-coding 场景，conc-list 为 [4, 8, 10, 12, 16, 20]，开启 DSpark 投机解码（spec-decoding: mtp，num-spec-tokens: 2，合成接受长度 2.51）。
• 接入 CI 路径：runners/launch_mi355x-amds.sh → benchmarks/multi_node/agentic/kimik3_fp4_mi355x_vllm.sh → kimik3_agg_pp_job.slurm → kimik3_fp4_mi355x_vllm_tp8pp2.sh（rank-0 OpenAI 前端 + agentic 回放；rank-1 --headless PP worker）。
• 修复 e2e bring-up 中发现的问题：
  • 用 --entrypoint bash 覆盖 vllm-openai-rocm 镜像的 ENTRYPOINT=vllm（否则服务脚本路径会被当成 vLLM CLI 参数）。
  • 在已知 host 根目录上解析权重，并在所有分配节点上校验后再启动（/model 为只读挂载；权重缺失时会挂上空目录，随后 hf download 因只读文件系统失败）。
• 将该 recipe 从先前的 spec-decoding: none（仅 STP）切换为 DSpark，此前已定位并修复 CONC=20 稳定性问题。
• 在 experimental/kimik3-v4/aiter/ 下新增运行时补丁：
  • patch_kv_offload_block_cap.py — 将 SimpleCPUOffloadConnector 的 CPU block 池钳制到最小 PP stage 容量（KV_OFFLOAD_MAX_CPU_BLOCKS），修复 CONC=20 时 hipMemcpyBatchAsync 的 SIGSEGV。
  • patch_kv_offload_batch_chunk.py — block-ID 范围诊断埋点（默认关闭：KV_OFFLOAD_BATCH_CHUNK_PATCH=0）。
  • 以及 smoke 验证路径使用的 aiter sitecustomize / GEMM 合并 / n6288 + ca_flush_sync 补丁栈。
• 将 MAX_NUM_BATCHED_TOKENS 从 2048 提升到 8192，适配该预填充占主导的 agentic 负载（平均输入长度约 135K，p95 约 455K token）；这是主要吞吐杠杆。
• 在 perf-changelog.yaml 末尾追加 e2e 接入与 DSpark 切换条目。

镜像： vllm/vllm-openai-rocm:nightly-cb8104839c141609d99f1254459ef3a4f1bd4263
模型： moonshotai/Kimi-K3
KV 卸载： SimpleCPUOffloadConnector（kv-offloading: dram，kv-offload-backend: vllm-simple，dram-utilization: 0.50 → 约 1499 GB/节点）
拓扑： 聚合 TP8 × PP2，ASYNC_SCHEDULING=false（可通过环境变量切换）

根因（CONC=20 SIGSEGV）

SimpleCPUOffloadScheduler 按全局 GPU KV 缓存配置分配共享 CPU block-ID 空间（约 47,095）。各 PP stage 的 connector worker 则按自身唯一 KV tensor 数量独立推导 pinned buffer（PP0: 3 → 更大；PP1: 5 → 约 28,257）。调度器一旦发出超过 PP1 buffer 的 ID，hipMemcpyBatchAsync 越界写 → PP1 全部 8 个 worker SIGSEGV。本仓库用 block-cap 补丁钳制到最小 stage 容量；上游修复：vllm-project/vllm#52921（草稿）。

Smoke 验证（g06+g17，30 分钟 profile，零 GPU 故障）

并发	单卡吞吐	TTFT p50	E2E p50	GPU KV 占用率
16	2,049 tok/s	—	—	—
20	2,078 tok/s	1,682 ms	8,963 ms	90.4%

测试计划

• 打上 full-sweep-fail-fast；在本 PR 的某个 commit 上跑出含 evals 的全绿 full sweep
• 确认启用 DSpark 后 c4/c8/c10/c12/c16/c20 均通过并成功上传产物
• 全绿且 evals 通过后评论 /reuse-sweep-run

相关

• 上游：vllm-project/vllm#52921
• PR：[AMD][AgentX] Kimi-K3 FP4 MI355X agentic TP8xPP2 DSpark / [AMD][AgentX] MI355X 上 Kimi-K3 FP4 agentic TP8×PP2 DSpark #2678