[nightly-verify] main is red

The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

Field	Value
`verify` job result	`failure`
`make verify-fmt`	`success`
`make verify-clippy`	`success`
`make verify-test`	`failure`
`make verify-test-video`	`success`
Trigger	`schedule`

Duration of the verify job: 16m2s.

Commit: 878f72a53564b2849bbe4ae0be1be818ca933582
Run: https://github.com/lablup/mlxcel/actions/runs/33666326398

An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

Reproduce locally with make verify (or the single failing target above).
This issue is reused by subsequent nightly reports of the same kind while it stays open.

The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

Field	Value
`verify` job result	`failure`
`make verify-fmt`	`success`
`make verify-clippy`	`success`
`make verify-test`	`failure`
`make verify-test-video`	`success`
Trigger	`schedule`

Duration of the verify job: 16m4s.

Commit: b2ff1eee96e6f30bdd3a5abb415498dde2f1067d
Run: https://github.com/lablup/mlxcel/actions/runs/33789607902

An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

Reproduce locally with make verify (or the single failing target above).
This issue is reused by subsequent nightly reports of the same kind while it stays open.

The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

Field	Value
`verify` job result	`failure`
`make verify-fmt`	`success`
`make verify-clippy`	`success`
`make verify-test`	`failure`
`make verify-test-video`	`success`
Trigger	`schedule`

Duration of the verify job: 14m48s.

Commit: 64e8d9a8b991e750fe58faff2b0e15cc377017fd
Run: https://github.com/lablup/mlxcel/actions/runs/33904935685

An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

Reproduce locally with make verify (or the single failing target above).
This issue is reused by subsequent nightly reports of the same kind while it stays open.

The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

Field	Value
`verify` job result	`failure`
`make verify-fmt`	`success`
`make verify-clippy`	`success`
`make verify-test`	`failure`
`make verify-test-video`	`success`
Trigger	`schedule`

Duration of the verify job: 16m3s.

Commit: 9bef3f2334c4048d8540aa392ef030c97d4dad2c
Run: https://github.com/lablup/mlxcel/actions/runs/33983424065

An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

Reproduce locally with make verify (or the single failing target above).
This issue is reused by subsequent nightly reports of the same kind while it stays open.

The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

Field	Value
`verify` job result	`failure`
`make verify-fmt`	`success`
`make verify-clippy`	`success`
`make verify-test`	`failure`
`make verify-test-video`	`success`
Trigger	`schedule`

Duration of the verify job: 9m1s.

Commit: bb8dd8240489c6c2702ac8c9870e78a3dafc5809
Run: https://github.com/lablup/mlxcel/actions/runs/34051078957

An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

Reproduce locally with make verify (or the single failing target above).
This issue is reused by subsequent nightly reports of the same kind while it stays open.

The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

Field	Value
`verify` job result	`failure`
`make verify-fmt`	`success`
`make verify-clippy`	`success`
`make verify-test`	`failure`
`make verify-test-video`	`success`
Trigger	`schedule`

Duration of the verify job: 9m6s.

Commit: bb8dd8240489c6c2702ac8c9870e78a3dafc5809
Run: https://github.com/lablup/mlxcel/actions/runs/34150956559

An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

Reproduce locally with make verify (or the single failing target above).
This issue is reused by subsequent nightly reports of the same kind while it stays open.

The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

Field	Value
`verify` job result	`failure`
`make verify-fmt`	`success`
`make verify-clippy`	`failure`
`make verify-test`	`success`
`make verify-test-video`	`success`
Trigger	`schedule`

Duration of the verify job: 9m17s.

Commit: 0accedd90ae9ea0679121bd087dafdd82882182a
Run: https://github.com/lablup/mlxcel/actions/runs/34262127778

An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

Reproduce locally with make verify (or the single failing target above).
This issue is reused by subsequent nightly reports of the same kind while it stays open.

The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

Field	Value
`verify` job result	`failure`
`make verify-fmt`	`success`
`make verify-clippy`	`success`
`make verify-test`	`failure`
`make verify-test-video`	`success`
Trigger	`schedule`

Duration of the verify job: 16m15s.

Commit: 86d35b2f25fde87172f429eb485206dfc23a0513
Run: https://github.com/lablup/mlxcel/actions/runs/34513246160

An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

Reproduce locally with make verify (or the single failing target above).
This issue is reused by subsequent nightly reports of the same kind while it stays open.

The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

Field	Value
`verify` job result	`failure`
`make verify-fmt`	`success`
`make verify-clippy`	`failure`
`make verify-test`	`success`
`make verify-test-video`	`success`
Trigger	`schedule`

Duration of the verify job: 10m17s.

Commit: 26b53f0246c3910e073f24310660569f1fbcc1c6
Run: https://github.com/lablup/mlxcel/actions/runs/36464262071

An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

Reproduce locally with make verify (or the single failing target above).
This issue is reused by subsequent nightly reports of the same kind while it stays open.

The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

Field	Value
`verify` job result	`failure`
`make verify-fmt`	`success`
`make verify-clippy`	`success`
`make verify-test`	`failure`
`make verify-test-video`	`success`
Trigger	`schedule`

Duration of the verify job: 18m23s.

Commit: d913fc7d737db98004c70c038c97b9454d4b3da7
Run: https://github.com/lablup/mlxcel/actions/runs/36611112322

An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

Reproduce locally with make verify (or the single failing target above).
This issue is reused by subsequent nightly reports of the same kind while it stays open.

The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

Field	Value
`verify` job result	`failure`
`make verify-fmt`	`success`
`make verify-clippy`	`success`
`make verify-test`	`failure`
`make verify-test-video`	`success`
Trigger	`schedule`

Duration of the verify job: 17m52s.

Commit: 78af3e2df9c8186be3f5ba71cb68c24527a2c1bd
Run: https://github.com/lablup/mlxcel/actions/runs/36757563932

An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

Reproduce locally with make verify (or the single failing target above).
This issue is reused by subsequent nightly reports of the same kind while it stays open.

The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

Field	Value
`verify` job result	`failure`
`make verify-fmt`	`success`
`make verify-clippy`	`success`
`make verify-test`	`failure`
`make verify-test-video`	`success`
Trigger	`schedule`

Duration of the verify job: 2m5s.

Commit: 78af3e2df9c8186be3f5ba71cb68c24527a2c1bd
Run: https://github.com/lablup/mlxcel/actions/runs/36905958058

An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

Reproduce locally with make verify (or the single failing target above).
This issue is reused by subsequent nightly reports of the same kind while it stays open.

The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

Field	Value
`verify` job result	`failure`
`make verify-fmt`	`success`
`make verify-clippy`	`success`
`make verify-test`	`failure`
`make verify-test-video`	`success`
Trigger	`schedule`

Duration of the verify job: 10m55s.

Commit: c0b71344c032177d495637422c7af50949d97fea
Run: https://github.com/lablup/mlxcel/actions/runs/37046630998

An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

Reproduce locally with make verify (or the single failing target above).
This issue is reused by subsequent nightly reports of the same kind while it stays open.

Triage of the Metal nightly red streak (M1 Ultra, 2026-09-28 to 10-05), read from the job logs.

Runs	Signature	Status
09-28 (`26b53f0`)	`verify-clippy` exit 101: unused `Result` in `fused_norm_parity_tests.rs:158` and `fused_rope_parity_tests.rs:133`	fixed by #2029, gone from 09-29
09-29 (`d913fc7`)	`tests::family_order_is_exhaustive`: `FAMILY_ORDER` missing `Speech`	fixed by #2080, gone from 09-30
09-30 to 10-05	`mxfp4/mxfp8_gather_qmm_matches_host_reference`, `SortedSharedActivation`	open, see below

Root cause of the remaining failure (high confidence, proven from source; not run on Metal): upstream MLX Metal GatherQMM::eval_gpu passes x.size() / K as the row count to gather_qmm_rhs, i.e. the rows of x before gather_qmm_rhs broadcasts x against the indices. The test case gathers one shared activation row [1, 1, K] against 32 sorted slots (M == 1, B = 32, E = 8, which hits the rhs path), so the grid and the kernel's M bound cover one row and output rows 1..31 stay as uninitialized malloc memory. That gives exactly 7.4e32 relative error (mxfp4) and non-finite values (mxfp8).

Not a test bug: the same runs pass mxfp_matmuls_match_host_reference_on_cpu_device, which runs this case on MLX's CPU backend, and the three earlier gather cases pass in bf16/f16/f32. "bf16" is only the first dtype tried.
Not a regression: the tests arrived with test(rocm): prove mxfp8 end to end, incl. FP8 block checkpoints #2071 on 09-30 and exposed a latent upstream edge case. ml-explore/mlx main still has the same line. Production MoE is unaffected, since every sorted SwitchLinear::forward caller passes gather_sort output with x already expanded to B rows.
The CUDA overlay computes M and B from out, which is why the test passes on CUDA.

Fix: draft PR #2138 passes B * M in the overlaid quantized.cpp. It is unverified on Metal and stays draft.

Next step (needs a Mac): cargo test --profile test-fast --features metal,accelerate -p mlxcel --lib models::switch_layers::mxfp_tests -- --test-threads=1 on the PR branch, plus one MoE prefill smoke over 64 slots, then mark #2138 ready. This issue closes when a nightly goes green.

perf(moe): share gather indices across SwitchGLU projections (#2137)

The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

Field	Value
`verify` job result	`failure`
`make verify-fmt`	`success`
`make verify-clippy`	`success`
`make verify-test`	`failure`
`make verify-test-video`	`success`
Trigger	`schedule`

Duration of the verify job: 22m9s.

Commit: b7116d1dae8a7c538973df89fd3a6aee5e032154
Run: https://github.com/lablup/mlxcel/actions/runs/37510161968

An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

Reproduce locally with make verify (or the single failing target above).
This issue is reused by subsequent nightly reports of the same kind while it stays open.

fix(metal): size gather_qmm_rhs by the broadcast x rows

The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

Field	Value
`verify` job result	`failure`
`make verify-fmt`	`success`
`make verify-clippy`	`success`
`make verify-test`	`failure`
`make verify-test-video`	`success`
Trigger	`schedule`

Duration of the verify job: 11m43s.

Commit: 1a0ff240c88aed27f912765534c126975e6c2678
Run: https://github.com/lablup/mlxcel/actions/runs/37665825551

An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

Reproduce locally with make verify (or the single failing target above).
This issue is reused by subsequent nightly reports of the same kind while it stays open.

The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

Field	Value
`verify` job result	`failure`
`make verify-fmt`	`success`
`make verify-clippy`	`success`
`make verify-test`	`failure`
`make verify-test-video`	`success`
Trigger	`schedule`

Duration of the verify job: 11m37s.

Commit: f715fe01f8c90aa540d74937cf28f604232122ab
Run: https://github.com/lablup/mlxcel/actions/runs/37823161843

An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

Reproduce locally with make verify (or the single failing target above).
This issue is reused by subsequent nightly reports of the same kind while it stays open.

The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

Field	Value
`verify` job result	`failure`
`make verify-fmt`	`success`
`make verify-clippy`	`success`
`make verify-test`	`failure`
`make verify-test-video`	`success`
Trigger	`schedule`

Duration of the verify job: 21m44s.

Commit: 25baebcc94e628fcfb691ee09e3ad25660e9f1ca
Run: https://github.com/lablup/mlxcel/actions/runs/37972246957

An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

Reproduce locally with make verify (or the single failing target above).
This issue is reused by subsequent nightly reports of the same kind while it stays open.

The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

Field	Value
`verify` job result	`failure`
`make verify-fmt`	`success`
`make verify-clippy`	`success`
`make verify-test`	`failure`
`make verify-test-video`	`success`
Trigger	`schedule`

Duration of the verify job: 18m46s.

Commit: 2807850767d0996019a36201d8654aa2ba14086a
Run: https://github.com/lablup/mlxcel/actions/runs/38075132911

An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

Reproduce locally with make verify (or the single failing target above).
This issue is reused by subsequent nightly reports of the same kind while it stays open.

The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

Field	Value
`verify` job result	`failure`
`make verify-fmt`	`success`
`make verify-clippy`	`success`
`make verify-test`	`failure`
`make verify-test-video`	`success`
Trigger	`schedule`

Duration of the verify job: 20m20s.

Commit: e24fb7b267cc9f82a8a5e2ed3e282d509b816dfb
Run: https://github.com/lablup/mlxcel/actions/runs/38163195615

An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

Reproduce locally with make verify (or the single failing target above).
This issue is reused by subsequent nightly reports of the same kind while it stays open.

macOS handoff for the 2026-10-12 issue triage: this issue remains with the Mac verification work, outside the GB10 auto-implementation batch. The latest failure report links run 38163195615. Check that run's first actual failure before attributing it to the older gather-QMM defect. PR #2138 already has M1 Ultra evidence in #2158, but this issue still requires a green post-merge Metal nightly. Coordinate with #2158 and #1986; no new Metal execution is claimed here.

The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

Field	Value
`verify` job result	`failure`
`make verify-fmt`	`success`
`make verify-clippy`	`failure`
`make verify-test`	`failure`
`make verify-test-video`	`success`
Trigger	`schedule`

Duration of the verify job: 32m31s.

Commit: 1e8c7022e197cd47dc833af74cd8ae5e5f78d5db
Run: https://github.com/lablup/mlxcel/actions/runs/38289904512

An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

Reproduce locally with make verify (or the single failing target above).
This issue is reused by subsequent nightly reports of the same kind while it stays open.

Navigation Menu

Repository navigation

[nightly-verify] main is red #1599

3 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[nightly-verify] main is red #1599

Description

Activity

github-actions commented on Sep 3, 2026

github-actions commented on Sep 4, 2026

github-actions commented on Sep 5, 2026

github-actions commented on Sep 6, 2026

github-actions commented on Sep 7, 2026

github-actions commented on Sep 8, 2026

github-actions commented on Sep 10, 2026

github-actions commented on Sep 28, 2026

github-actions commented on Sep 29, 2026

github-actions commented on Sep 30, 2026

github-actions commented on Oct 1, 2026

github-actions commented on Oct 2, 2026

3 remaining items

inureyes commented on Oct 6, 2026

github-actions commented on Oct 6, 2026

github-actions commented on Oct 7, 2026

github-actions commented on Oct 8, 2026

github-actions commented on Oct 9, 2026

github-actions commented on Oct 10, 2026

github-actions commented on Oct 11, 2026

inureyes commented on Oct 12, 2026

github-actions commented on Oct 12, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions