[NV][AgentX] Add glm5.1-fp8-b200-tilert-agentic (TileRT PD disagg) / [NV][AgentX] 新增 glm5.1-fp8-b200-tilert-agentic（TileRT PD 分离） - #2650
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
89289b1 to
a344d06
Compare
|
Claude finished @Oseltamivir's task in 14m 25s —— View job PR Review — glm5.1-fp8-b200-tilert-agentic (TileRT PD disagg)
SummaryNo blocking issues found. Two 🟡 warnings posted inline:
Everything else checked out: changelog entry is appended at the physical tail with the master-config edit paired; images are public ( |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| if [[ "$TILERT_IS_AGENTIC" == "1" ]]; then | ||
| run_agentic_replay; BENCH_RC=$? | ||
| else | ||
| run_bench_and_eval; BENCH_RC=$? | ||
| fi |
There was a problem hiding this comment.
🟡 WARNING: On a failed replay, the script aborts here before touch "$DONE_SENTINEL" because of a leaked set -e.
Why it matters: run_agentic_replay_and_write_outputs (benchmark_lib.sh:2194–2214) toggles set +e/set -e and returns with errexit left enabled. run_node.sh does not run under set -e and its shutdown contract relies on always reaching touch "$DONE_SENTINEL" (line 301) so the decode node exits gracefully. Inside run_agentic_replay, the failing call is caught by || { rc=$?; ... }, but per bash semantics the leaked set -e takes effect once that AND-OR list completes — so return $rc with a nonzero rc makes the plain call run_agentic_replay; BENCH_RC=$? exit the whole script immediately. The sentinel is never touched, the decode node never sees the shutdown signal, and submit.sh only force-kills the decode srun after the TILERT_DECODE_DRAIN timeout. The non-agentic path is unaffected (nothing on it enables errexit), so this is new behavior introduced with the agentic branch.
Fix: call the function in an AND-OR list (which suppresses errexit for the call) and restore the script's no-errexit state afterwards:
| if [[ "$TILERT_IS_AGENTIC" == "1" ]]; then | |
| run_agentic_replay; BENCH_RC=$? | |
| else | |
| run_bench_and_eval; BENCH_RC=$? | |
| fi | |
| if [[ "$TILERT_IS_AGENTIC" == "1" ]]; then | |
| run_agentic_replay && BENCH_RC=0 || BENCH_RC=$? | |
| { set +e; } 2>/dev/null | |
| else | |
| run_bench_and_eval; BENCH_RC=$? | |
| fi |
| - config-keys: | ||
| - glm5.1-fp8-b200-tilert-agentic |
There was a problem hiding this comment.
🟡 WARNING: This PR also changes the runtime of the existing glm5.1-fp8-b200-tilert config, but the changelog entry only lists the new agentic key.
Why it matters: Three changes in this diff affect the existing fixed-seq-len config, not just the agentic one:
benchmarks/multi_node/tilert_utils/setup_deps.shbumps the defaultTILERT_VERSIONfrom0.1.5.post2to0.1.5.post3(installed on both roles for every TileRT run);configs/nvidia-master.yamlbumps the existing key'srouter.versionto0.1.5.post3;run_node.shnow always passes--queue-timeout "$TILERT_QUEUE_TIMEOUT"(0for non-agentic) topd_router, a flag the old invocation never sent.
The post-merge sweep re-benchmarks only the keys listed in the appended changelog entry, so glm5.1-fp8-b200-tilert's published results will remain the 0.1.5.post2 numbers while its master-config entry claims 0.1.5.post3 — stale provenance for that curve.
Fix: add the existing key to this entry (with a description line for the TileRT pin bump and the new --queue-timeout 0 router flag), e.g.:
| - config-keys: | |
| - glm5.1-fp8-b200-tilert-agentic | |
| - config-keys: | |
| - glm5.1-fp8-b200-tilert-agentic | |
| - glm5.1-fp8-b200-tilert |
If the intent is that --queue-timeout 0 and post3 are behavior-identical for the fixed-seq-len path, that's still a dependency-version change to a published config and should be re-swept or explicitly documented.
| fi | ||
| TILERT_QUEUE_TIMEOUT=${TILERT_QUEUE_TIMEOUT:-0} | ||
|
|
||
| AGENTIC_LOGS_DIR=${AGENTIC_LOGS_DIR:-$RESULT_DIR/LOGS/agentic} |
There was a problem hiding this comment.
🔴 run_node.sh:36 defaults AGENTIC_LOGS_DIR to $RESULT_DIR/LOGS/agentic, and RESULT_DIR defaults to /workspace (the GITHUB_WORKSPACE bind-mount). run_agentic_replay() then mkdir -p's $AGENTIC_LOGS_DIR/conc_ for each concurrency point, creating new directories under /workspace, which violates the non-negotiable AGENTS.md invariant that benchmarks create no new directories there. Since the tilert container runs as root under enroot with no user remap, this can leave root-owned dirs in the shared runner checkout that block a later job's git checkout -- the fix is to default AGENTIC_LOGS_DIR outside /workspace, the same way the sibling benchmarks/multi_node/agentic_srt.sh defaults its result dir to /logs/agentic.
Extended reasoning...
The bug: run_node.sh line 36 sets AGENTIC_LOGS_DIR=${AGENTIC_LOGS_DIR:-$RESULT_DIR/LOGS/agentic}, and RESULT_DIR defaults to /workspace at line 8 (pre-existing). Neither variable is overridden anywhere else along the tilert launch chain (launch_b200-dgxc.sh -> the new glm5.1_fp8_b200_tilert-disagg.sh -> submit.sh -> run_node.sh), so by default AGENTIC_LOGS_DIR resolves to /workspace/LOGS/agentic.
Where it triggers: the new run_agentic_replay() function loops over CONC_LIST and does mkdir -p "$AGENTIC_LOGS_DIR/conc_${conc}" for every concurrency point. With this PR's config (conc-list: [1]), that is one call producing /workspace/LOGS/agentic/conc_1, but the code generalizes to any list. /workspace inside the container is bind-mounted from $GITHUB_WORKSPACE on the shared Slurm/enroot runner (per submit.sh's --container-mounts), i.e. it's the CI checkout directory, not a scratch dir.
Why nothing else prevents it: the pre-existing non-agentic path (run_bench_and_eval) only ever writes result files flat into RESULT_DIR via --result-dir; it never creates subdirectories, so this class of violation didn't exist in the tilert script before this PR. This is a genuinely new violation introduced here, not a latent pre-existing one.
Why it's forbidden: AGENTS.md line 24 states a non-negotiable benchmark invariant: 'Benchmarks create no new directories under /workspace. Root containers must not leave root-owned files in shared ... runner workspaces.' The tilert containers run under enroot with no user namespace remap (the existing convert_weights/TILERT_WEIGHTS_DIR logic in this same file is deliberately routed to /lustre/fsw/gharunners/... instead of /workspace for exactly this reason), so any directory this script creates under /workspace will be root-owned in the shared GITHUB_WORKSPACE checkout on the dgxc-slurm runner.
Impact: a root-owned /workspace/LOGS/agentic/conc_1 directory left behind after the job can block a subsequent job's git checkout/clean on the same shared runner checkout -- exactly the failure mode the invariant exists to prevent, and the same class of problem launch_b200-dgxc.sh already works around elsewhere (e.g. its NFS silly-rename cleanup loop, and routing model conversion outputs away from /workspace).
Proof (step-by-step):
submit.shmounts$GITHUB_WORKSPACE:/workspaceinto the container.run_node.shruns withRESULT_DIRunset -> defaults to/workspace(line 8).- Line 36:
AGENTIC_LOGS_DIRunset -> defaults to/workspace/LOGS/agentic. - For the new
glm5.1-fp8-b200-tilert-agenticscenario,TILERT_ROLE=prefillcallsrun_agentic_replay(). - For
conc=1(the sole point inconc-list: [1]):conc_result_dir=/workspace/LOGS/agentic/conc_1;mkdir -pcreates it, owned by root (container runs as root, no remap). - This directory persists in the shared
GITHUB_WORKSPACEcheckout after the job exits, and a later job's checkout/clean on the same runner can fail against the root-owned tree.
The fix: mirror the sibling agentic recipe. benchmarks/multi_node/agentic_srt.sh:14 does BASE_RESULT_DIR=${RESULT_DIR:-/logs/agentic} -- outside /workspace -- precisely to avoid this. run_node.sh:36 should default AGENTIC_LOGS_DIR the same way, e.g. AGENTIC_LOGS_DIR=${AGENTIC_LOGS_DIR:-/logs/agentic} (or route it under the same out-of-workspace tree already used for TILERT_WEIGHTS_DIR), rather than nesting it under RESULT_DIR.
| run_agentic_replay() { | ||
| wait_for_server_ready --port "$ROUTER_PORT" \ | ||
| --server-log "$BENCHMARK_LOGS_DIR/tilert_router.log" --server-pid "$ROUTER_PID" | ||
| local rc=0 conc conc_result_dir | ||
| local result_filename_base="$RESULT_FILENAME" | ||
| for conc in $CONC_LIST; do | ||
| conc_result_dir="$AGENTIC_LOGS_DIR/conc_${conc}" | ||
| mkdir -p "$conc_result_dir" | ||
| export CONC="$conc" | ||
| export RESULT_FILENAME="${result_filename_base}_conc${conc}" | ||
| build_replay_cmd "$conc_result_dir" | ||
| run_agentic_replay_and_write_outputs "$conc_result_dir" \ | ||
| || { rc=$?; echo "[agentic] WARNING: conc=$conc failed/timed out (rc=$rc)"; } | ||
| done | ||
| export RESULT_FILENAME="$result_filename_base" | ||
| return $rc | ||
| } | ||
|
|
There was a problem hiding this comment.
🟡 run_agentic_replay() in tilert_utils/run_node.sh:245-262 is a third copy of the same concurrency-loop scaffolding (loop CONC_LIST → mkdir conc_ → export CONC/RESULT_FILENAME → build_replay_cmd → run_agentic_replay_and_write_outputs) already duplicated in agentic_srt.sh and amd_utils/trace_replay.sh. The shared per-request primitives already live in benchmark_lib.sh, but this outer loop/naming contract does not, so a future change to it has to be hand-applied in three places.
Extended reasoning...
What: run_agentic_replay() (new in this PR, benchmarks/multi_node/tilert_utils/run_node.sh:245-262) loops over $CONC_LIST, creates a conc_<N> result directory, exports CONC/RESULT_FILENAME, and calls the shared build_replay_cmd + run_agentic_replay_and_write_outputs helpers. This is the third time this exact scaffolding is written out — benchmarks/multi_node/agentic_srt.sh (~lines 104-120) and benchmarks/multi_node/amd_utils/trace_replay.sh (~lines 130-155) both implement the identical loop/mkdir/export/call sequence around the same two benchmark_lib.sh primitives.
Code path: All three copies share the same contract: iterate CONC_LIST, create .../conc_${conc}, export CONC=$conc, set RESULT_FILENAME to a conc-suffixed name, call build_replay_cmd, then run_agentic_replay_and_write_outputs. build_replay_cmd (benchmark_lib.sh:1958) and run_agentic_replay_and_write_outputs (benchmark_lib.sh:2187) are already factored out as shared primitives, but the outer loop/directory/naming contract that wires them together is not — it lives independently in each of the three call sites.
Why existing code doesn't prevent it: benchmark_lib.sh already centralizes the per-request primitives, so there was an established place to add a shared wrapper; this PR instead adds a fresh copy inline in run_node.sh rather than extending or calling into a common helper. Nothing enforces that the three copies stay in sync — amd_utils/trace_replay.sh even carries a comment noting it must mirror agentic_srt.sh's contract by hand.
Impact: Low. The three copies currently agree on the conc_ directory + _conc filename suffix contract, and each has legitimate small variations (agentic_srt.sh does an idle-wait between points; amd_utils/trace_replay.sh clears KV caches between points). Nothing is broken today. But a future change to the per-conc directory/result-naming contract (e.g. adding a manifest file, changing the suffix format) would need to be hand-applied in three places, and a missed update would silently produce inconsistent result layouts across backends rather than fail loudly.
Suggested fix: Extract the common loop (iterate CONC_LIST, mkdir the per-conc dir, export CONC/RESULT_FILENAME, call build_replay_cmd, call run_agentic_replay_and_write_outputs, handle rc) into a shared helper in benchmark_lib.sh, parameterized by the two behavior hooks (idle-wait, cache-clear) via optional callback env vars or function names. Each of the three call sites would then just supply their environment and hooks.
Proof by construction: Diff the three implementations line-by-line: (1) run_node.sh:250-258 — for conc in $CONC_LIST; do conc_result_dir="$AGENTIC_LOGS_DIR/conc_${conc}"; mkdir -p ...; export CONC="$conc"; export RESULT_FILENAME="..."; build_replay_cmd ...; run_agentic_replay_and_write_outputs ...; done. (2) agentic_srt.sh ~104-120 — same shape, same five steps, plus one extra call to wait_for_agentic_servers_idle. (3) amd_utils/trace_replay.sh ~130-155 — same shape again, plus one extra call to clear_kv_caches. Removing the one extra line from each of (2) and (3) leaves three byte-for-byte identical loops — that identity is the DRY violation.
| export UCX_NET_DEVICES="${UCX_NET_DEVICES:-mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1}" | ||
| export UCX_MEMTYPE_CACHE="${UCX_MEMTYPE_CACHE:-n}" | ||
| export UCX_MEMTYPE_REG_WHOLE="${UCX_MEMTYPE_REG_WHOLE:-n}" | ||
| TILERT_DISAGG="$GITHUB_WORKSPACE/benchmarks/multi_node/${EXP_NAME%%_*}_${PRECISION}_b200_${FRAMEWORK}-disagg.sh" | ||
| TILERT_SUBDIR="multi_node" | ||
| [[ "${SCENARIO_SUBDIR}" == "agentic/" ]] && TILERT_SUBDIR="multi_node/agentic" | ||
| TILERT_DISAGG="$GITHUB_WORKSPACE/benchmarks/${TILERT_SUBDIR}/${EXP_NAME%%_*}_${PRECISION}_b200_${FRAMEWORK}-disagg.sh" | ||
| [[ -f "$TILERT_DISAGG" ]] || { echo "tilert disagg script not found: $TILERT_DISAGG"; exit 1; } | ||
| exec bash "$TILERT_DISAGG" |
There was a problem hiding this comment.
🟡 sweep:AIPERF_MMAP_CACHE_HOST_PATH|HF_HUB_CACHE_HOST_PATH -- the tilert multinode IS_AGENTIC path never receives the shared aiperf/HF caches every other agentic recipe gets. Example: runners/launch_b200-dgxc.sh's tilert branch (line 145) execs into $TILERT_DISAGG and exits before the later 'if [[ "$IS_AGENTIC" == "1" ]]' block that mounts $AIPERF_MMAP_CACHE_HOST_PATH and $HF_HUB_CACHE_HOST_PATH; fix by also wiring those two into benchmarks/multi_node/tilert_utils/submit.sh's run_role() --container-mounts/--export so the tilert container gets /aiperf_mmap_cache and /hf_hub_cache with matching env vars, same as the srtctl and single-node paths.
Extended reasoning...
What the bug is
runners/launch_b200-dgxc.sh mounts and exports the two shared Lustre-backed caches — AIPERF_MMAP_CACHE_HOST_PATH (aiperf's content-addressed trace mmap cache) and HF_HUB_CACHE_HOST_PATH (the shared HF hub cache) — only in two places: the single-node srun --container-mounts=...,$AIPERF_MMAP_CACHE_HOST_PATH:/aiperf_mmap_cache branch near the bottom of the file, and the srtctl multinode if [[ "$IS_AGENTIC" == "1" ]] block that builds DEFAULT_MOUNTS_BLOCK and writes it into srtslurm.yaml's default_mounts.
The new glm5.1-fp8-b200-tilert-agentic scenario (added by this PR) goes through neither of those. It hits the IS_MULTINODE == "true" branch for FRAMEWORK == "tilert" at runners/launch_b200-dgxc.sh:138-145, which resolves $TILERT_DISAGG and does exec bash "$TILERT_DISAGG"; exit 1. exec replaces the current shell process, so control never reaches the later IS_AGENTIC block further down in the same script — that block is structurally unreachable for any tilert run, agentic or not.
Code path and why nothing else catches it
From there, TILERT_DISAGG (benchmarks/multi_node/agentic/glm5.1_fp8_b200_tilert-disagg.sh) execs into benchmarks/multi_node/tilert_utils/submit.sh, whose run_role() sets --container-mounts="$GITHUB_WORKSPACE:/workspace,$MODEL_PATH:$MODEL_PATH,$TILERT_WEIGHTS_DIR:$TILERT_WEIGHTS_DIR" and --export=ALL,TILERT_ROLE=...,DECODE_HOST=...,PREFILL_HOST=...,PORT=... — no aiperf cache mount, no HF hub cache mount, and neither AIPERF_DATASET_MMAP_CACHE_DIR nor HF_HUB_CACHE is exported. This PR's own run_node.sh changes add a TILERT_IS_AGENTIC prefill-side call to resolve_trace_source (which does hf download --repo-type dataset for cc-traces-weka-062126-256k, intending to land in the shared HF cache per its own comment) and build_replay_cmd (which builds aiperf's dataset mmap cache) — both landing in ephemeral, per-container default locations instead of the persistent mount every other agentic recipe uses.
Impact
Every job for this scenario re-downloads the multi-GB trace dataset and rebuilds AIPerf's mmap cache from scratch inside its enroot container, instead of reusing the Lustre-backed caches that both the single-node and srtctl-multinode agentic paths already share across jobs. The run still succeeds (the author's validation reports 539 requests, zero errors) — this is a persistent efficiency/reuse gap, not a correctness break.
Step-by-step proof
- Scenario
glm5.1-fp8-b200-tilert-agenticsetsFRAMEWORK=tilert,multinode: true,scenarios.agentic-coding→ the launcher takesIS_MULTINODE=="true"andFRAMEWORK=="tilert". runners/launch_b200-dgxc.sh:145runsexec bash "$TILERT_DISAGG"— process image replaced, script exits via the exec'd process's exit code; theIS_AGENTICmount block later in the file is dead code for this path.submit.sh run_role()'s--container-mounts/--export(lines 55, 57) contain no reference toAIPERF_MMAP_CACHE_HOST_PATH,HF_HUB_CACHE_HOST_PATH,AIPERF_DATASET_MMAP_CACHE_DIR, orHF_HUB_CACHE.- Inside the container,
run_node.sh's prefill role callsresolve_trace_sourceandbuild_replay_cmdwith those env vars unset, so both the dataset download and the mmap cache build go to ephemeral container-local storage, discarded when the job's enroot container is torn down. - Next job run: cache miss again, full re-download and rebuild — unlike single-node and srtctl-multinode agentic runs, which persist both caches on Lustre.
Fix
Wire AIPERF_MMAP_CACHE_HOST_PATH and a shared HF_HUB_CACHE_HOST_PATH into tilert_utils/submit.sh's run_role() --container-mounts (as /aiperf_mmap_cache and /hf_hub_cache) and add AIPERF_DATASET_MMAP_CACHE_DIR=/aiperf_mmap_cache and HF_HUB_CACHE=/hf_hub_cache to its --export, mirroring what the srtctl DEFAULT_MOUNTS_BLOCK and the single-node srun invocation already do.
a344d06 to
cd778c3
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32160853729 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32162401646 |
cd778c3 to
809a733
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32164342913 |
…a TileRT) Ported from #2645. Original PR author: CrimsonDump (@CrimsonDump). Build and pin an internal TileRT post2 queueing backport because upstream post3 is unavailable. 从 #2645 迁移。原 PR 作者：CrimsonDump（@CrimsonDump）。 由于上游 post3 不可获取，构建并固定内部 TileRT post2 排队 backport。 (cherry picked from commit 24d7457)
809a733 to
5eb5760
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32209559996 |
2 participants
Attribution / 作者署名
This internal PR reproduces #2645 from the repository branch
agent/internalize-pr-2645because the original head branch is hosted in a fork.All implementation credit belongs to the original PR author, @CrimsonDump (XieBaijie). The imported commit preserves CrimsonDump as its author.
本内部 PR 从仓库内分支
agent/internalize-pr-2645复现 #2645，因为原 PR 的 head 分支位于 fork 中。全部实现贡献归原 PR 作者 @CrimsonDump（XieBaijie）所有；导入后的 commit 保留 CrimsonDump 为作者。
Description
Add the GLM-5.1-FP8 B200 AgentX (
agentic-coding) submission on TileRT PD disaggregation as one new master-config key and one new multi-node recipe.glm5.1-fp8-b200-tilert-agentic: vLLM prefill + TileRT decode, 1P1D with TP8 on each side and NIXL KV transfer.cluster:b200-dgxcrunner label.[1].benchmarks/multi_node/agentic/throughSCENARIO_SUBDIR.LOGS/agentic/conc_<N>/and accept the Hugging Face repository ID as a served model name.0.1.5.post2+inferencex.1wheel and wait up to 1,800 seconds for the single decode node instead of returning HTTP 429 immediately.中文说明
新增 GLM-5.1-FP8 B200 AgentX（
agentic-coding）在 TileRT PD 分离上的提交，包含一个新的主配置键和一个新的多节点配方。glm5.1-fp8-b200-tilert-agentic：vLLM prefill + TileRT decode，1P1D、两侧各 TP8，KV 通过 NIXL 传输。cluster:b200-dgxcrunner label。[1]。SCENARIO_SUBDIR将 agentic 配置路由到benchmarks/multi_node/agentic/。LOGS/agentic/conc_<N>/下新增 AIPerf trace replay，并接受 Hugging Face repository ID 作为 served model name。0.1.5.post2+inferencex.1wheel；单个 decode node 忙时最多等待 1,800 秒，不再立即返回 HTTP 429。Internal wheel / 内部 wheel
The public TileRT repository cannot build the native wheel and upstream
post3is unavailable. This PR therefore preserves the officialpost2native libraries and patches onlytilert/pd_vllm/pd_router.py.benchmarks/multi_node/tilert_utils/build_queue_wheel.pybenchmarks/multi_node/tilert_utils/patches/tilert-0.1.5.post2-queue.patch4c0a4330b96d3cb96536d761197e978bde18030bb1e3d14f2a39472053ab4b7bTileRT 公共仓库无法构建 native wheel，且上游
post3不可获取。因此，本 PR 保留官方post2native library，仅修改tilert/pd_vllm/pd_router.py。benchmarks/multi_node/tilert_utils/build_queue_wheel.pybenchmarks/multi_node/tilert_utils/patches/tilert-0.1.5.post2-queue.patch4c0a4330b96d3cb96536d761197e978bde18030bb1e3d14f2a39472053ab4b7bValidation / 验证
Python 3.12 rebuilt the wheel successfully; archive,
RECORD, metadata, local version, and patched-router parity checks passed.Threaded tests covered immediate acquisition, bounded waiting, release notification, timeout, and async event-loop offload.
Anonymous download through the exact launcher URL reproduced the pinned SHA256.
YAML parsing and Bash syntax passed for all touched files.
Exact-key generation produced one AgentX job on
cluster:b200-dgxc, concurrency[1], router0.1.5.post2+inferencex.1.validate_perf_changelog.pypassed against currentorigin/mainwith additions only.pytest utils/matrix_logic/: 231 passed.Run 32164342913 established the post2 failure mode: 3 of 27 profiling requests received HTTP 429 (
all decode nodes busy), exceeding AIPerf's 10% threshold. The replacement run validates this backport.Python 3.12 成功重建 wheel；archive、
RECORD、metadata、local version 与 patched-router parity 检查均通过。线程测试覆盖立即获取、有界等待、release notification、timeout 与 async event-loop offload。
通过 launcher 使用的精确 URL 匿名下载，得到固定 SHA256。
所有修改文件的 YAML 解析和 Bash 语法检查通过。
精确配置生成得到一个 AgentX job：
cluster:b200-dgxc、并发[1]、router0.1.5.post2+inferencex.1。validate_perf_changelog.py对当前origin/main校验通过，仅有新增内容。pytest utils/matrix_logic/：231 项通过。Run 32164342913 证明了 post2 的失败模式：27 个 profiling request 中 3 个收到 HTTP 429（
all decode nodes busy），超过 AIPerf 10% 阈值。replacement run 将验证本 backport。Remaining observation / 剩余观察项
TileRT does not expose router/decode Prometheus metrics, and GLM-5.1's 202752-position context limit leaves 175 of 393 traces eligible.
TileRT 未暴露 router/decode Prometheus metrics；GLM-5.1 的 202752-position context limit 使 393 条 trace 中有 175 条符合条件。