Skip to content

[AMD] [AGENTX] Qwen3.5 FP4 MI355X SGLang Agentic MTP / Qwen3.5 FP4 MI355X SGLang Agentic 数据集 MTP 基准配置 #2605

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ChangLiu0709 wants to merge 9 commits into from
Closed

[AMD] [AGENTX] Qwen3.5 FP4 MI355X SGLang Agentic MTP / Qwen3.5 FP4 MI355X SGLang Agentic 数据集 MTP 基准配置 #2605

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
4bd577b
amd/agentx-qwen3.5-fp4: add non-MTP agentic config for MI355X
ChangLiu0709 Aug 14, 2026
3caa99e
Enable EAGLE MTP speculative decoding for Qwen3.5 FP4 agentic config
ChangLiu0709 Aug 14, 2026
155d2a2
Use GLM-5.2 MTP settings: num-steps=5, draft-tokens=6, acc-len=3.61
ChangLiu0709 Aug 14, 2026
775b0bc
Align conc-list and kv-offloading with GLM-5.2 agentic MTP config
ChangLiu0709 Aug 14, 2026
4ebaf8e
Use TP4/EP4 single arm matching qwen3.5-fp4-mi355x-sglang-agentic-mtp
ChangLiu0709 Aug 14, 2026
6702d3e
Merge remote-tracking branch 'origin/main' into chang/qwen3.5-fp4-mi3…
ChangLiu0709 Aug 14, 2026
42b55a7
perf-changelog: add entry for qwen3.5-fp4-mi355x-sglang-agentic
ChangLiu0709 Aug 14, 2026
0a09bbf
Merge origin/main: resolve perf-changelog.yaml conflict (append-only)
ChangLiu0709 Aug 17, 2026
fd04201
Merge remote-tracking branch 'origin/main' into chang/qwen3.5-fp4-mi3…
ChangLiu0709 Aug 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 129 additions & 0 deletions benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
#!/usr/bin/env bash
set -euo pipefail
set -x

# AgentX trace replay for Qwen3.5-397B-A17B MXFP4 on MI355X with SGLang
# native EAGLE MTP speculative decoding. Throughput uses the committed
# golden synthetic acceptance length; evaluation retains real target-model
# verification.

source "$(dirname "$0")/../../benchmark_lib.sh"

export EVAL_FRAMEWORK="lm-eval"

check_env_vars \
MODEL TP CONC EP_SIZE RESULT_DIR DURATION

SCHEDULER_RECV_INTERVAL=${SCHEDULER_RECV_INTERVAL:-30}

if [[ -n "${SLURM_JOB_ID:-}" ]]; then
echo "JOB $SLURM_JOB_ID running on ${SLURMD_NODENAME:-unknown}"
fi

if [[ -n "${MODEL_PATH:-}" ]]; then
if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then
hf download "$MODEL" --local-dir "$MODEL_PATH"
fi
else
hf download "$MODEL"
export MODEL_PATH="$MODEL"
fi

rocm-smi || true
amd-smi || true

export WEKA_LOADER_OVERRIDE=semianalysis_cc_traces_weka_062126_256k
resolve_trace_source
install_agentic_deps

export AIPERF_SERVER_METRICS_URLS="http://localhost:${PORT}/metrics"
export AIPERF_REQUIRED_SERVER_METRIC_PREFIX="sglang:"

SERVER_LOG="$RESULT_DIR/server.log"
mkdir -p "$RESULT_DIR"

SERVER_PID=""
cleanup_agentic_services() {
local exit_code=$?
trap - EXIT INT TERM
set +e
stop_background_process_tree "$SERVER_PID" "SGLang server" 60
exit "$exit_code"
}
trap cleanup_agentic_services EXIT
trap 'exit 130' INT
trap 'exit 143' TERM

PARALLEL_ARGS=(
--tp "$TP"
--dp 1
--ep-size "$EP_SIZE"
)

TOKENIZER_ARGS=()
if [ "$TP" -ge 4 ]; then
TOKENIZER_ARGS=(--tokenizer-worker-num 6)
fi

MAX_RUNNING_REQUESTS=$((2 * CONC))
CUDA_GRAPH_MAX_BS="$CONC"
[ "$CUDA_GRAPH_MAX_BS" -gt 64 ] && CUDA_GRAPH_MAX_BS=64

export PYTHONNOUSERSITE=1
export SGLANG_USE_AITER=1
export SGLANG_USE_AITER_UNIFIED_ATTN=1
export AITER_FLYDSL_FORCE=1
export SGLANG_MAMBA_SSM_DTYPE=bfloat16
export SGLANG_TIMEOUT_KEEP_ALIVE=1800

if [ "${EVAL_ONLY:-false}" != "true" ]; then
export SGLANG_SIMULATE_ACC_LEN=3.61
export SGLANG_SIMULATE_ACC_METHOD=match-expected
export SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token
fi

SGLANG_CMD=(
python3 -m sglang.launch_server
--model-path "$MODEL_PATH"
--served-model-name "$MODEL"
--host 0.0.0.0
--port "$PORT"
--trust-remote-code
"${PARALLEL_ARGS[@]}"
--attention-backend aiter
--mem-fraction-static 0.80
--model-loader-extra-config '{"enable_multithread_load": true}'
--watchdog-timeout 1200
--page-size 16
--cuda-graph-max-bs "$CUDA_GRAPH_MAX_BS"
--max-running-requests "$MAX_RUNNING_REQUESTS"
--max-prefill-tokens 32768
--chunked-prefill-size 32768
--scheduler-recv-interval "$SCHEDULER_RECV_INTERVAL"
--stream-interval 50
"${TOKENIZER_ARGS[@]}"
--tokenizer-path "$MODEL"
--reasoning-parser qwen3
--tool-call-parser qwen3_coder
--speculative-algorithm EAGLE
--speculative-num-steps 5
--speculative-eagle-topk 1
--speculative-num-draft-tokens 6
--enable-metrics
--enable-cache-report
)

printf '%q ' "${SGLANG_CMD[@]}" | tee "$RESULT_DIR/sglang_command.txt"
printf '\n' | tee -a "$RESULT_DIR/sglang_command.txt"
"${SGLANG_CMD[@]}" > "$SERVER_LOG" 2>&1 &
SERVER_PID=$!

wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID"

if [ "${EVAL_ONLY:-false}" = "true" ]; then
run_eval --port "$PORT"
else
build_replay_cmd "$RESULT_DIR"
REPLAY_CMD+=" --apply-chat-template"
run_agentic_replay_and_write_outputs "$RESULT_DIR"
fi
17 changes: 15 additions & 2 deletions configs/amd-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -361,6 +361,20 @@ qwen3.5-fp4-mi355x-sglang-mtp:
- { tp: 2, conc-start: 4, conc-end: 128, spec-decoding: mtp }
- { tp: 4, conc-start: 4, conc-end: 16, spec-decoding: mtp }

qwen3.5-fp4-mi355x-sglang-agentic:
image: lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260811
model: amd/Qwen3.5-397B-A17B-MXFP4
model-prefix: qwen3.5
runner: cluster:mi355x-amds
precision: fp4
framework: sglang
multinode: false
scenarios:
agentic-coding:
- dram-utilization: 0.80
search-space:
- { tp: 4, ep: 4, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [1, 2, 4, 8, 10, 12, 16] }

@claude claude Bot Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 BLOCKING: This new config is an exact functional duplicate of qwen3.5-fp4-mi355x-sglang-agentic-mtp (line 403), and the new script benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang.sh is dead code that will never run.

Why it matters: The launcher derives the script name from SPEC_DECODING, not the config key — runners/launch_mi355x-amds.sh:260 sets SPEC_SUFFIX=_mtp whenever spec-decoding: mtp, and lines 307–314 then resolve benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang_mtp.sh, which exists and wins. Since this arm sets spec-decoding: mtp, this config runs the old _mtp.sh script (num-steps=3, draft-tokens=4, SGLANG_SIMULATE_ACC_LEN=3.39) — the advertised num-steps=5 / draft-tokens=6 / AL=3.61 settings live only in the unreachable new script. Net effect: two config keys run the byte-identical benchmark (same image, model, TP4/EP4, HiCache, conc-list, same script), doubling GPU sweep time and publishing duplicate results, while none of the PR's stated MTP changes take effect.

Fix: Apply the intended changes (num-steps 5, draft-tokens 6, SGLANG_SIMULATE_ACC_LEN=3.61) directly to benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang_mtp.sh, and delete this duplicate config key plus the new qwen3.5_fp4_mi355x_sglang.sh script. (Per MODELS.md, agentic recipes ship spec-decode-only, so a separate non--mtp agentic key isn't needed — and a key named -agentic whose only arm is MTP is misleading either way.)

@claude claude Bot Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Both new/updated search-space arms in configs/amd-master.yaml (qwen3.5-fp4-mi355x-sglang-agentic line 389 and qwen3.5-fp4-mi355x-sglang-agentic-mtp line 401) set spec-decoding: mtp, which makes runners/launch_mi355x-amds.sh route both to the pre-existing qwen3.5_fp4_mi355x_sglang_mtp.sh (num-steps=3, draft-tokens=4, acc-len=3.39) instead of the new qwen3.5_fp4_mi355x_sglang.sh this PR adds (num-steps=5, draft-tokens=6, acc-len=3.61, HiCache). The new script's suffix-less filename is unreachable dead code under this launcher convention; rename it with a _mtp suffix (replacing the stale script) so both recipes actually pick up the intended EAGLE settings.

Extended reasoning...

The bug. runners/launch_mi355x-amds.sh resolves which benchmark script to run purely from a filename convention: SPEC_SUFFIX=$([[ "$SPEC_DECODING" == "mtp" ]] && printf '_mtp' || printf '') (line 260), then SCRIPT_FW="benchmarks/single_node/${SCENARIO_SUBDIR}${SCRIPT_BASE}_${FRAMEWORK}${SPEC_SUFFIX}.sh" (line 307-308), preferring SCRIPT_FW whenever it exists on disk (line 310). SCRIPT_BASE derives from EXP_NAME%%_*, which for both Qwen3.5 agentic configs is qwen3.5, giving SCRIPT_BASE=qwen3.5_fp4_mi355x.

Both arms this PR touches — the brand-new qwen3.5-fp4-mi355x-sglang-agentic (line 389) and the rewritten qwen3.5-fp4-mi355x-sglang-agentic-mtp (line 401) — set spec-decoding: mtp on their sole search-space entry. That forces SPEC_SUFFIX=_mtp for both, so SCRIPT_FW resolves to benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang_mtp.sh in every case. I confirmed on disk that this file already exists and is untouched by this PR: it has --speculative-num-steps 3, --speculative-num-draft-tokens 4, and SGLANG_SIMULATE_ACC_LEN=3.39 (lines 108/110/79 of that file).

The new artifact never runs. This PR's actual new file, benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang.sh (no _mtp suffix), contains the intended settings — --speculative-num-steps 5, --speculative-num-draft-tokens 6, SGLANG_SIMULATE_ACC_LEN=3.61 — plus the HiCache KV-offload wiring described in the PR summary. Per the launcher's naming convention, that file is only ever selected when SPEC_DECODING != mtp. Neither arm in either config satisfies that condition (both explicitly set spec-decoding: mtp), so the new script is unreachable dead code — it will sit in the repo but never execute in a real sweep.

Step-by-step proof:

  1. configs/amd-master.yaml:389 (new qwen3.5-fp4-mi355x-sglang-agentic): search-space arm is { tp: 4, ep: 4, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [...] }.
  2. generate_sweep_configs.py builds EXP_NAME starting with the model code qwen3.5 and exports SPEC_DECODING=mtp for this arm.
  3. launch_mi355x-amds.sh:260: SPEC_DECODING=="mtp" → SPEC_SUFFIX=_mtp.
  4. launch_mi355x-amds.sh:307-308: SCRIPT_BASE=qwen3.5_fp4_mi355x; SCRIPT_FW=benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang_mtp.sh.
  5. That file exists on disk (pre-existing, unmodified by this PR) → line 310 picks it as BENCHMARK_SCRIPT.
  6. The new file qwen3.5_fp4_mi355x_sglang.sh (no _mtp) is only reachable via SCRIPT_FW when SPEC_SUFFIX is empty, i.e. SPEC_DECODING != mtp — never true for either config's arm.
  7. Result: both qwen3.5-fp4-mi355x-sglang-agentic and qwen3.5-fp4-mi355x-sglang-agentic-mtp execute the stale script with num-steps=3/draft-tokens=4/acc-len=3.39, not the PR's intended num-steps=5/draft-tokens=6/acc-len=3.61 with HiCache.

Impact. Every sweep run under either config key silently measures the old, unrelated speculative-decoding configuration rather than what the PR claims to add — acceptance-length and throughput numbers will not reflect the EAGLE MTP tuning or HiCache offloading the PR is meant to introduce, and the committed new script is permanently unreachable dead code as written.

Fix. Rename benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang.sh to qwen3.5_fp4_mi355x_sglang_mtp.sh (replacing the stale pre-existing file with the same name), so the _mtp suffix the launcher expects actually resolves to the intended script. Alternatively, drop spec-decoding: mtp from both arms if a non-suffixed script were truly the goal, but that contradicts the PR description's explicit MTP intent, so the rename is the correct fix.


Comment on lines +364 to +377

@claude claude Bot Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 This PR adds a brand-new recipe (qwen3.5-fp4-mi355x-sglang-agentic in configs/amd-master.yaml plus a new benchmark script) but does not append the required entry to perf-changelog.yaml. Per AGENTS.md's non-negotiable benchmark invariant, every recipe addition needs a new append-only tail entry — please add one describing this config before merging.

Extended reasoning...

AGENTS.md, under Non-negotiable benchmark invariants, states verbatim: 'Every change that can affect benchmark performance and every recipe addition or modification requires a new perf-changelog.yaml entry. The file is append-only and byte-sensitive... append only at the tail.' This is not a stylistic suggestion — it's called out as a non-negotiable invariant that gates recipe changes in this repository.

This PR introduces a brand-new top-level config key, qwen3.5-fp4-mi355x-sglang-agentic, in configs/amd-master.yaml (lines 377-391), along with its own new benchmark script benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang.sh. This is unambiguously a 'recipe addition' as defined by the invariant — it's a new model/hardware/framework combination (Qwen3.5-397B MXFP4 baseline, non-MTP, on MI355X via SGLang) with its own search space (TP2/EP2 and TP4/EP1 arms) and its own image pin.

The PR diff touches exactly two files: the new benchmark script and configs/amd-master.yaml. perf-changelog.yaml is not part of the diff at all. I confirmed this directly by reading the tail of the current perf-changelog.yaml in the repo: the last two entries are for PR #2578 (dsv4-fp4-b200-sglang-agentic-hicache-mtp) and PR #2577 (dsv4-fp4-b300-sglang-agentic-hicache-mtp) — there is no entry referencing PR #2605 or the qwen3.5-fp4-mi355x-sglang-agentic config key anywhere in the file.

Step-by-step proof:

  1. Read AGENTS.md — non-negotiable invariant requires a perf-changelog.yaml entry for every recipe addition, appended at the tail.
  2. Read the PR diff — a new config key qwen3.5-fp4-mi355x-sglang-agentic is added to configs/amd-master.yaml, and a new script implements it. This satisfies the definition of 'recipe addition.'
  3. Enumerate the PR's changed files — only 2 files: the script and the config. perf-changelog.yaml is absent.
  4. Read the tail of perf-changelog.yaml on the base branch — ends with the PR Refresh DeepSeek-V4 B300 SGLang AgentX MTP #2577/Refresh DeepSeek-V4 B200 SGLang AgentX MTP #2578 entries, with no entry for this recipe or PR [AMD] [AGENTX] Qwen3.5 FP4 MI355X SGLang Agentic MTP / Qwen3.5 FP4 MI355X SGLang Agentic 数据集 MTP 基准配置 #2605.
  5. Conclusion: the required changelog entry is missing.

Nothing in the diff suggests this recipe is exempt (e.g. it isn't a config-only tweak to an existing key — it's a wholly new entry with a companion script). The fix is straightforward: append a new entry at the tail of perf-changelog.yaml with config-keys: [qwen3.5-fp4-mi355x-sglang-agentic], scenario-type: [agentic-coding], a short description of the TP2/EP2 and TP4/EP1 non-MTP baseline, and the PR link for #2605 — following the exact format of the adjacent entries already in the file.

Comment thread
claude[bot] marked this conversation as resolved.
qwen3.5-fp4-mi355x-sglang-agentic-mtp:
image: lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260811
model: amd/Qwen3.5-397B-A17B-MXFP4
Expand All @@ -373,8 +387,7 @@ qwen3.5-fp4-mi355x-sglang-agentic-mtp:
agentic-coding:
- dram-utilization: 0.80
search-space:
- { tp: 2, ep: 2, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 12, 16, 20] }
- { tp: 4, ep: 1, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 12, 16, 20, 24, 28, 32, 40] }
- { tp: 4, ep: 4, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [1, 2, 4, 8, 10, 12, 16], spec-decoding: mtp }

@claude claude Bot Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 BLOCKING: Both arms now declare kv-offloading: dram with the HiCache backend, but the script that actually runs (benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang_mtp.sh — see routing comment above; the new qwen3.5_fp4_mi355x_sglang.sh has the same gap) contains no HiCache handling at all: no agentic_kv_offload_enabled / require_agentic_kv_offload_backend hicache check and no --enable-hierarchical-cache / --hicache-* server flags.

Why it matters: benchmark_lib.sh only validates that KV_OFFLOADING / KV_OFFLOAD_BACKEND / TOTAL_CPU_DRAM_GB are set (benchmarks/benchmark_lib.sh:78-104); it does not add any server flags. So the sweep will pass validation and launch SGLang without a host DRAM KV tier, while the results are labeled and published as HiCache DRAM offload — mislabeled benchmark data. Compare the GLM-5.2 recipe this PR says it mirrors: benchmarks/single_node/agentic/glm5.2_fp4_mi355x_sglang_mtp.sh:86-149 builds CACHE_ARGS (--enable-hierarchical-cache --hicache-ratio ... --hicache-write-policy ... --hicache-io-backend ... --hicache-mem-layout ...) gated on agentic_kv_offload_enabled and passes them to sglang.launch_server.

Fix: Add an equivalent CACHE_ARGS block (gated on agentic_kv_offload_enabled / require_agentic_kv_offload_backend hicache) to the script this config actually routes to, and include "${CACHE_ARGS[@]}" in SGLANG_CMD — or set kv-offloading: none if HiCache wasn't actually part of the tuned/validated runs.


qwen3.5-fp4-mi355x-sglang-disagg:
image: lmsysorg/sglang-rocm:v0.5.12.post1-rocm720-mi35x-20260523
Expand Down
11 changes: 11 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6049,3 +6049,14 @@
- "Add GB200 GLM-5.2 FP4 Dynamo-SGLang AgentX with EAGLE MTP, HiCache DRAM offload, and KV-aware correlation-ID affinity."
- "Measure the TP8 aggregate curve and tuned 1P1D TP4 HiCache c10/c12 disaggregated frontier with every logical SGLang metrics endpoint."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2620

- config-keys:
- qwen3.5-fp4-mi355x-sglang-agentic
- qwen3.5-fp4-mi355x-sglang-agentic-mtp
scenario-type:
- agentic-coding
description:
- "Add new qwen3.5-fp4-mi355x-sglang-agentic config for Qwen3.5-397B-A17B MXFP4 on MI355X with native EAGLE MTP (num-steps=5, draft-tokens=6, topk=1), TP4/EP4, HiCache DRAM KV offloading, and golden simulated acceptance length 3.61."
- "Consolidate qwen3.5-fp4-mi355x-sglang-agentic-mtp from two arms (TP2/EP2 + TP4/EP1, no KV offloading) to a single TP4/EP4 arm with HiCache DRAM KV offloading, aligned concurrency sweep [1, 2, 4, 8, 10, 12, 16]."
- "New benchmark script benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang.sh with EAGLE MTP, HiCache, qwen3 reasoning parser, and qwen3_coder tool-call parser."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2605
Loading