chore(ci): link an OpenXLA binary in CI so link-only regressions cannot reach main

[inureyes]

Summary

xla-compile runs cargo check, which never invokes the linker, so a regression in the IREE link recipe in build.rs can reach main unobserved; #1274 was exactly that failure, found by hand after the fact rather than by CI. This PR adds a new xla-link job that actually links an OpenXLA target on the self-hosted GB10 runner, closing the gap xla-compile's own comment already recorded as open.

What changed

• changes job: a new xla_link output from a sibling dorny/paths-filter filter keyed on build.rs, src/lib/mlxcel-xla/build.rs, src/lib/mlxcel-xla/csrc/**, scripts/iree/**, rust-toolchain.toml, and .github/workflows/ci.yml. That is narrower than the existing rust filter, so the link job does not run on every Rust PR, but it covers the whole causal surface of a link failure: the root build.rs holds the IREE recipe, scripts/iree/** pins the distribution whose archive set the recipe names, mlxcel-xla's build script and its csrc/** sources produce the shim object whose undefined symbols those archives resolve, and rust-toolchain.toml is on the path because fix: Integration tests cannot link with --features cuda,xla-iree #1274 was entirely about where rustc places its own -lc relative to appended rustc-link-arg entries.
• New xla-link job, placed immediately after xla-compile, needs: changes, gated on github.repository == 'lablup/mlxcel' && needs.changes.outputs.xla_link == 'true', runs-on: GB10, permissions: contents: read, timeout-minutes: 120, a job-scoped concurrency group keyed per PR with cancel-in-progress, and its own CARGO_TARGET_DIR ($HOME/.cargo-target/mlxcel-xla-link-ci) separate from xla-compile's and from the release job's.
• The link step runs cargo test --release --features cuda,xla-iree --test xla_prepared_prefill --no-run. --release is mandatory because the debug profile cannot link these targets on this host at all (hundreds of relocation truncated to fit: R_AARCH64_CALL26 errors against ordinary libstd symbols, the unoptimized binary exceeding the AArch64 direct-branch range). --no-run links without executing, so the job never touches the GPU. Because cargo also builds the package's [[bin]] targets whenever an integration test is selected, this command links mlxcel, mlxcel-server, speculative_bench, and mlxcel-bench-decode in addition to the test binary, so it is a superset of the --bin mlxcel-server alternative rather than a cheaper substitute for it. RUSTFLAGS is deliberately left unset, unlike xla-compile, so a red run here is unambiguously a link failure rather than a lint failure.
• The concurrency group exists because xla-link is the first job on the shared GB10 runner whose warm cost is minutes rather than seconds. Without it, three pushes to a build.rs PR queue three link jobs of up to 120 minutes each on the one runner that also serves cargo-clippy for every Rust PR and the release build.
• Comment blocks recording the trigger rationale, the measured cost, what the gate was demonstrated on, and the residual coverage gaps: the IREE_DIST and macOS IREE_MACOS_HOME recipes in build.rs remain unverified because no runner in this repository holds either distribution, and a link regression arriving through a path outside the filter is still not caught at PR time.
• Corrected the note above the clippy job, which asserted that the repository guard means fork PRs never queue on the self-hosted runner. It does not. On a pull_request event opened from a fork against this repository, github.repository is the base repository, so the guard is true and the job runs here. The guard's real effect is to stop the job queueing forever in a fork of the repository. What gates the fork-PR case is the repository's Actions fork-PR approval policy. That reading applies equally to xla-compile and to the new job, and the issue specifying this work repeated the incorrect version, so it was propagating.

Verification

All of the following ran on the GB10 host with the IREE runtime provisioned (bash scripts/iree/setup-cuda.sh, eval "$(bash scripts/iree/setup-cuda.sh --env)", MLX_CUDA_ARCHITECTURES=121).

• The job's link command passes from a purged CARGO_TARGET_DIR: exit 0 in 15m40s, ending at Executable tests/xla_prepared_prefill.rs. Most of that is MLX's CUDA sources compiling from scratch, not the link.
• Warm cost, which is what a build.rs edit actually pays because it invalidates the mlxcel crate but not mlxcel-core's MLX build: 6m01s and 7m24s across two runs, both exit 0.
• The gate catches a link-only regression that xla-compile cannot. With -l:libflatcc_parsing.a dropped from the IREE_CUDA_HOME branch of build.rs, xla-compile's command cargo check --features cuda,xla-iree --all-targets still exits 0 in 59 seconds, while this job's command exits 101 with error: linking with cc failed and undefined references to flatcc_verify_string_field, flatcc_verify_table_vector_field, and flatcc_verify_vector_field. Restoring the archive returns the link to exit 0. That is the same shape as fix: Integration tests cannot link with --features cuda,xla-iree #1274, on the same tree, and is the demonstration the issue asked for.
• The job runs green on this PR itself, since the PR changes .github/workflows/ci.yml and so matches the filter.

build.rs is unchanged by this PR; every control above was run outside the branch and reverted.

On the negative control the issue specified

The issue's acceptance criteria asked for the demonstration to be done by reverting the -lc entry added in #1275. That control was tried first and did not fail the link: the command exits 0 with rustc-link-arg=-lc confirmed absent from the emitted build-script output and the IREE archives confirmed present on the link line.

Why it does not fail is unresolved, and nothing here should be read as evidence that the entry is dead. An earlier revision of this PR claimed the pinned runtime was built without the stack protector; that claim was wrong and has been removed. It came from running nm in a shell where nm is an alias for an unrelated command, so the search silently matched nothing. Against the pinned distribution, using /usr/bin/nm: __stack_chk_guard is undefined in 176 objects of libiree_runtime_unified.a, including the call.c.o named in #1274; it is undefined in libc.so.6; and it is defined only in ld-linux-aarch64.so.1. Every precondition the build.rs comment records still holds today. Two further explanations were tested and ruled out: -lpthread and -ldl resolve to stub archives on this glibc rather than to scripts that would pull libc in late, and libm.so groups only libm.so.6.

So build.rs is left alone, the flatcc control above is what demonstrated the gate, and the workflow comment records all of this so the next reader does not delete the -lc entry on the strength of one control that failed to reproduce.

Closes #1303

[inureyes]

Security and performance review

The diff is one workflow file, so this is a CI supply chain and self-hosted-runner review rather than an application-code one. No CRITICAL or HIGH issue is introduced by this PR. One HIGH-severity property of the workflow is inherited unchanged from clippy and xla-compile; it is described below and is explicitly not a finding against this change.

Verified clean

• No script injection. Neither run: block in the new job interpolates ${{ github.* }} or any other untrusted context. Both use only shell variables and literal text, so the discipline the cross-repo-refs job documents (base ref passed through env:) is not violated here.
• permissions: contents: read on the job, under a top-level contents: read. This repository's default workflow permission is write, so the explicit narrowing is doing real work.
• persist-credentials: false on checkout, so no token is left in .git/config for the PR's own build.rs or scripts/iree/setup-cuda.sh to read.
• Trigger is pull_request, not pull_request_target, so fork code never runs against base-repo secrets or the base ref.
• --no-run really does not execute the linked binary, so the "never touches the GPU, never contends with development work" claim holds. The contention this job creates is runner occupancy, not GPU occupancy.
• Provisioning is not an open supply chain hole. setup-cuda.sh verifies the compiler wheel by sha256 and pins the runtime to an exact iree-org commit.
• Action pinning matches the rest of the file (actions/checkout@v7). Nothing here is SHA-pinned and sha_pinning_required is off repository-wide, so this is inherited, not introduced.
• The separate CARGO_TARGET_DIR is the right call. It is distinct from mlxcel-cuda-gb10 and mlxcel-cuda-aarch64, so a CI gate cannot invalidate or poison a release build's cache.

HIGH, inherited unchanged, not introduced by this PR

Fork pull requests do reach the self-hosted runner. github.repository == 'lablup/mlxcel' evaluates true on a PR opened from a fork against this repository, because on that event github.repository is the base repository. The guard's real effect is to stop the job queueing forever in a fork of the repository, which has no GB10 runner. The repository is public with forking enabled, its Actions fork-PR approval policy is first_time_contributors, and the runner runs as a personal user account whose home directory holds .ssh, .config/gh, and the model store. So a contributor with one merged PR can get code execution there through build.rs.

This PR adds no exposure. clippy and xla-compile are gated on rust == 'true' (**/*.rs), which is strictly wider than xla_link (build.rs, scripts/iree/**, rust-toolchain.toml, ci.yml), and both already run this repository's setup-cuda.sh and the PR's build.rs on the same runner. Any fork PR that could trigger xla-link already triggers both of them today.

Worth a separate hardening issue rather than scope creep here: adding && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) to all three GB10 jobs, tightening the approval policy to all external contributors, or giving the runner its own unprivileged account.

What I did fix (ab432ac7) is the comment above clippy that asserted the opposite ("Self-hosted, so fork PRs never queue on it"). #1303 repeated that reading when specifying this job, so it was actively propagating.

MEDIUM, amplified by this PR, fixed in ab432ac7

ci.yml has no concurrency control anywhere. That was tolerable while every GB10 job was seconds warm (clippy 27 to 40s, xla-compile about a minute); xla-link is 6m to 7m30s warm with timeout-minutes: 120, a step change. Without a group, three pushes to a build.rs PR queue three link jobs on the single runner that also serves clippy for every Rust PR and the release build, and the superseded ones run to completion instead of being cancelled. Added a job-scoped group keyed on the PR number, so superseded runs are cancelled and no other job's behaviour changes. Same pattern pipeline-parallel-ci.yml already uses.

MEDIUM, introduced here, not fixed

This adds a fourth permanent CI-owned target directory on a volume that is 93% full (282G free of 3.7T; ~/.cargo-target is already 39G, of which mlxcel-xla-ci alone is 18G). A release-profile link directory will grow well past its current 2.6G, and nothing prunes any of them. Not a blocker, but the same host runs releases, so a retention step or a follow-up issue is worth having.

LOW

• GITHUB_ENV injection surface, inherited verbatim from xla-compile: setup-cuda.sh --env | sed 's/^export //' >> "$GITHUB_ENV" would let a multi-line value set arbitrary variables for later steps (RUSTFLAGS, LD_PRELOAD, PATH). The emitted values are $IREE_DIR and $IREE_COMPILE, and the script is in-repo anyway, so this is redundant with simply editing the script. Noted, not changed.
• A green run does not prove a link happened. This PR's own OpenXLA feature link check passed in 12s with the link step taking 1s, reusing artifacts the manual GB10 validation left in the same directory. That is correct cargo behaviour, and the fingerprint does move for build.rs, rust-toolchain.toml, and an IREE_VERSION bump, which are the triggers that matter. The narrow hole: a scripts/iree/** edit that changes how the runtime is built without moving IREE_VERSION leaves both IREE_CUDA_HOME and build.rs unchanged, and the script's idempotence means the archives are not rebuilt either, so the job can green without relinking. Probably worth one line in the "Deliberately NOT covered" list; not worth a design change.
• The filter omits Cargo.toml and Cargo.lock, so a dependency pulling in a conflicting native library is not caught at PR time. Already recorded in the job's own comment, and widening it would fire on every dependency bump, so the tradeoff as written looks right.

Verdict

No unresolved CRITICAL or HIGH attributable to this PR. The two fixes above are in ab432ac7; the remaining items are either inherited posture or follow-up issues.

[inureyes]

Implementation Review Summary

Intent

Add a CI job that actually links an OpenXLA target so a regression in the IREE link recipe cannot reach main the way #1274 did, path-filtered, release profile, no GPU work.

Findings addressed in 225586be

• (HIGH) The -lc paragraph recorded a false fact. It stated the pinned IREE runtime "is built without the stack protector", that libiree_runtime_unified.a "holds no __stack_chk_guard reference at all (nm reports zero)", and that the -lc entry "is therefore inert". Against the pinned distribution on the GB10 host (~/.cache/mlxcel/iree-cuda-3.12.0rc20260721), /usr/bin/nm reports U __stack_chk_guard in 176 objects of that archive, including the call.c.o named in fix: Integration tests cannot link with --features cuda,xla-iree #1274's error. nm -D on libc.so.6 shows the symbol undefined there, and ld-linux-aarch64.so.1 is where it is defined. Those are exactly the preconditions build.rs states. The likely source of the zero reading is environmental: in the maintainer's interactive shell nm is an alias to a mosh command, so nm <archive> | grep -c ... returns 0 with no output at all. The comment now records that the -lc control did not reproduce, that the reason is unresolved, and that the flatcc control is what demonstrated the gate, without concluding the entry is dead.
• (MEDIUM) "What it links" was inaccurate about the binaries. Cargo builds the package's [[bin]] targets whenever an integration test is selected, so cargo test --test xla_prepared_prefill --no-run links mlxcel, mlxcel-server, speculative_bench, and mlxcel-bench-decode as well as the test. The validation run's target directory holds all four (test binary at 20:20:49, the four binaries at 20:21:59 to 20:23:59, same run). The rejected cargo build --bin mlxcel-server alternative is therefore a strict subset of the chosen command, not a costlier one. The chosen command is still the right one, since it links the integration test that fix: Integration tests cannot link with --features cuda,xla-iree #1274 failed on, but the rationale now says what actually happens and notes that the measured figures already include the binaries.
• (HIGH) The path filter missed part of the causal surface. src/lib/mlxcel-xla/build.rs compiles the C shim and emits rustc-link-lib=static=xla_iree plus its search path, and src/lib/mlxcel-xla/csrc/** produces the object whose undefined symbols the IREE archives resolve. A change there that leaves an unresolved symbol passes cargo check and, under the original filter, would not have started this job either: build.rs as a picomatch pattern matches only the root build script, and csrc/** matches no other filter in the file. Both paths are now in the xla_link filter and the trigger comment names them.

Verified and left alone

• xla_link output wiring, the sibling dorny/paths-filter entry, and the if: condition all parse and evaluate as intended (the inner filter document is valid YAML; the output name matches the filter key).
• Step shape matches xla-compile exactly: actions/checkout@v7 with persist-credentials: false, the CARGO_TARGET_DIR export into GITHUB_ENV, and the sed 's/^export //' idiom on the --env output.
• Its own CARGO_TARGET_DIR, so no cache contention with xla-compile or the release job. --no-run means no GPU work.
• The four settled design decisions (release profile, which target, path-filtered trigger, RUSTFLAGS unset) are sound as decisions; only the recorded reasoning was corrected.

Remaining item for the author

• The PR body still carries the disproved -lc narrative ("nm reports zero", "the -lc entry is therefore inert", and the suggested follow-up to weaken the build.rs comment). Since this repository preserves the PR body as the squash commit body, that text becomes permanent history on merge. Please correct it, and re-run the -lc control with an absolute nm path before drawing any conclusion about that entry.