Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -445,6 +445,8 @@ jobs:
persist-credentials: false
- name: Check every fused-kernel launcher routes through select_kernel_port
run: python3 scripts/ci/check_kernel_port_dispatch.py
- name: Check the port-dispatch checker rejects an unvalidated wave64 port
run: bash scripts/ci/check_kernel_port_dispatch_test.sh

binary-assets:
name: binary assets
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ Thank you for your interest in contributing to mlxcel! This document covers the
```
PR-time CI in [`ci.yml`](.github/workflows/ci.yml) runs far more than `cargo fmt`, `cargo deny`, and a crate-version check: a `changes` job path-filters most of the rest by what the PR touches, and `crate-versions`, `kernel-dtype-keys`, `binary-assets`, `cuda-arch-lists`, `license-headers`, `llama-compat-manifest`, and `webui-contract` need no toolchain, run in seconds, and gate every PR unconditionally. `cargo fmt` and `cargo deny` gate on Rust-touching PRs, as before, and so does clippy again: after #21/#23 dropped it for cost, #1285 re-added it as the `clippy` job (`cargo clippy -p mlxcel --lib --tests -- -D warnings`, default features) on the shared self-hosted `GB10` runner, where a persistent target directory keeps a warm run to 27-40 seconds; it fires whenever a Rust file changed, or the PR carries the `ci:full` label. That gated slice is narrower than the full `--workspace --all-targets --features metal,accelerate` command above, and the general unit suite stays unenforced (no PR job runs `cargo test --workspace`), so run both commands above yourself before you push. `GB10` also runs further path-gated jobs, each of which the `ci:full` label forces as well: `xla-compile` on Rust changes, `xla-link` on the IREE link-recipe paths only, `cuda-sm70-compile` and `cuda-blockfloat` on the CUDA-architecture paths (the `src/lib/mlx-cpp/` tree minus its ROCm and Metal subtrees, plus the MLX build scripts) rather than on ordinary Rust changes, and `webui-installed-artifact` on Rust or WebUI changes; the hosted `webui-bundle` job gates on the WebUI path filter alone (which includes the server and CLI sources), and the hosted `mlx-pin` job on the pinned-MLX-commit paths. `rocm-build` is parked on a `rocm-gfx1151` self-hosted runner until that runner is registered and `vars.ROCM_CI_ENABLED` is set; `rocm-ci-status` reports that parked state as an advisory warning, never a failing check. `workflow-lint` (`actionlint`) and the advisory `self-hosted-gate-advisory` both fire on workflow-file changes, the latter noting the path-gated `GB10`/`rocm-gfx1151` jobs above were not re-run unless the PR carries `ci:full`. `cross-repo-refs` runs on every PR but only advises (`STRICT=1` gates it locally). Outside `ci.yml`, [`pipeline-parallel-ci.yml`](.github/workflows/pipeline-parallel-ci.yml) still runs its own path-filtered clippy and a `distributed::`-scoped `cargo test` on pipeline-parallel changes. [`nightly-verify.yml`](.github/workflows/nightly-verify.yml) runs the full `make verify` once a day on `self-hosted-macos-26-arm64` and files an issue when `main` goes red or the run does not finish, a backstop and not a substitute. The exhaustive CUDA and ROCm test suites run in no PR job; `make verify-test-cuda` and `make verify-rocm` remain their only gates, run locally.

`make verify` runs eight prerequisite targets: `verify-versions`, `verify-kernel-dtype-keys`, `verify-kernel-port-dispatch`, `verify-llama-compat`, `verify-rocm-overlay`, `verify-fmt`, `verify-clippy`, and `verify-test`. The last three are the fmt, clippy, and test commands above; `cargo deny check`, the fourth command in that block, is not one of them, and PR CI is what gates it. The first five are cheap consistency gates: `verify-versions` asserts every version-tracking workspace crate carries the root `mlxcel` version, `verify-kernel-dtype-keys` asserts every CUDA and HIP JIT kernel launch keys its cache on the input dtypes and that the set of launcher files has not shrunk unnoticed (#1053, #1054, #1875), `verify-kernel-port-dispatch` asserts every fused-kernel launcher chooses its port through `select_kernel_port` (#1801, #1885), `verify-llama-compat` validates the checked-in llama-server b10621 compatibility manifest under `compat/llama-server/b10621/` structurally, with no network access required (#1443), and `verify-rocm-overlay` checks, offline, that the ROCm overlay's records agree with the MLX pin and the tree, so a pin bump that skips the ROCm retarget fails on any host (#1813). The nightly invokes those same Makefile targets, so the local gate and CI cannot drift apart. Tests build under `[profile.test-fast]` rather than `--release`: `opt-level = 3` is kept so MLX numerics stay representative, while cross-crate LTO and the single codegen unit that make full `--release` test links expensive are dropped. `make test-fast` / `make test-fast-cuda` are the same profile with `--test-threads=1` and a `FILTER` hook for narrowing the run while you iterate. Reach for `cargo test --release --features metal,accelerate` by hand only when you suspect a defect specific to release codegen. See [`docs/installation.md`](docs/installation.md#fast-iteration-builds) for the measured comparison.
`make verify` runs eight prerequisite targets: `verify-versions`, `verify-kernel-dtype-keys`, `verify-kernel-port-dispatch`, `verify-llama-compat`, `verify-rocm-overlay`, `verify-fmt`, `verify-clippy`, and `verify-test`. The last three are the fmt, clippy, and test commands above; `cargo deny check`, the fourth command in that block, is not one of them, and PR CI is what gates it. The first five are cheap consistency gates: `verify-versions` asserts every version-tracking workspace crate carries the root `mlxcel` version, `verify-kernel-dtype-keys` asserts every CUDA and HIP JIT kernel launch keys its cache on the input dtypes and that the set of launcher files has not shrunk unnoticed (#1053, #1054, #1875), `verify-kernel-port-dispatch` asserts every fused-kernel launcher chooses its port through `select_kernel_port` (#1801, #1885), and that only HIP ports with no lane-level operation are marked to run on wave64 ROCm devices (#2147), `verify-llama-compat` validates the checked-in llama-server b10621 compatibility manifest under `compat/llama-server/b10621/` structurally, with no network access required (#1443), and `verify-rocm-overlay` checks, offline, that the ROCm overlay's records agree with the MLX pin and the tree, so a pin bump that skips the ROCm retarget fails on any host (#1813). The nightly invokes those same Makefile targets, so the local gate and CI cannot drift apart. Tests build under `[profile.test-fast]` rather than `--release`: `opt-level = 3` is kept so MLX numerics stay representative, while cross-crate LTO and the single codegen unit that make full `--release` test links expensive are dropped. `make test-fast` / `make test-fast-cuda` are the same profile with `--test-threads=1` and a `FILTER` hook for narrowing the run while you iterate. Reach for `cargo test --release --features metal,accelerate` by hand only when you suspect a defect specific to release codegen. See [`docs/installation.md`](docs/installation.md#fast-iteration-builds) for the measured comparison.

The plain `make test`, `make test-verbose`, and `make test-lib` targets stay on Cargo's default `dev` profile and now export `RUST_MIN_STACK=16777216` for you. That stack bump is there for `server::reasoning_effort_tests`: its repeated renders of the pinned 8,952-byte Qwen3.8 chat template can overflow libtest's default 2 MiB per-test thread stack under unoptimized `dev`, even though the same suite passes under `test-fast` and `release`. If you bypass the Makefile and run the `dev`-profile server tests directly, keep the same prefix yourself: `RUST_MIN_STACK=16777216 cargo test --lib --features metal,accelerate server::`. `make verify-test` is intentionally unaffected because it already builds under `[profile.test-fast]`.

Expand Down
7 changes: 6 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -822,10 +822,15 @@ verify-kernel-dtype-keys: ## Assert every CUDA and HIP JIT kernel launch keys it
@python3 scripts/ci/check_kernel_dtype_keys.py
@bash scripts/ci/check_kernel_dtype_keys_test.sh

# check_kernel_port_dispatch_test.sh is the negative coverage for the ROCm
# wavefront rules (#2147): it marks a shuffle-based table any-wave, adds a
# shuffle to a marked body and calls the wave-size test seam from production
# code in a throwaway copy, and asserts the checker rejects each.
.PHONY: verify-kernel-port-dispatch
verify-kernel-port-dispatch: ## Assert every fused-kernel launcher chooses its port through select_kernel_port (issues #1801, #1885)
verify-kernel-port-dispatch: ## Assert every fused-kernel launcher chooses its port through select_kernel_port, and only lane-free HIP ports run on wave64 (issues #1801, #1885, #2147)
@echo "$(CYAN)[verify] kernel port dispatch...$(RESET)"
@python3 scripts/ci/check_kernel_port_dispatch.py
@bash scripts/ci/check_kernel_port_dispatch_test.sh

# Offline half of the mlxcelverse ROCm overlay checks (issue #1813). It reads
# only files in this repository: patches-rocm/UPSTREAM must name the build's MLX
Expand Down
Loading
Loading