chore(ci): link an OpenXLA binary in CI so link-only regressions cannot reach main #1303

Problem / Background

The xla-compile job added by PR #1282 (.github/workflows/ci.yml:308-352) catches compile errors under the OpenXLA feature combinations but cannot catch link errors, because cargo check never invokes the linker. The job's own exclusion list at .github/workflows/ci.yml:296-307 records this gap; this issue is the work of closing it.

That gap is not hypothetical. Issue #1274 was exactly a link failure, and cargo check --features cuda,xla-iree --all-targets passed cleanly on the same tree that could not link a single integration test:

/usr/bin/ld: libiree_runtime_unified.a(call.c.o): undefined reference to symbol '__stack_chk_guard@@GLIBC_2.17'
/usr/bin/ld: /lib/ld-linux-aarch64.so.1: error adding symbols: DSO missing from command line

It reached main and was found by hand while validating an unrelated PR. The fix, PR #1275, appends a second -lc after the IREE archives in build.rs (see the comment block at build.rs:143-176, ending with the "-lc" entry in the --start-group list), because rustc emits its own -lc before anything cargo:rustc-link-arg can append.

Current Behavior

.github/workflows/ci.yml:348-352 runs two commands on the self-hosted GB10 runner:

cargo check --features cuda,xla-iree --all-targets
cargo check --no-default-features --features xla-diagnostics --all-targets

Neither links. Nothing else in CI compiles any XLA feature at all: pipeline-parallel-ci.yml runs clippy on default features, nightly-verify.yml runs make verify with metal,accelerate, and release.yml does not build the xla-iree feature. So a regression in the link recipe in build.rs reaches main unobserved and is caught only when a developer builds locally with these features.

Proposed Solution

Add a job to .github/workflows/ci.yml that links at least one target with --features cuda,xla-iree on the GB10 runner, reusing the xla-compile job's provisioning shape: bash scripts/iree/setup-cuda.sh followed by bash scripts/iree/setup-cuda.sh --env | sed 's/^export //' >> "$GITHUB_ENV", MLX_CUDA_ARCHITECTURES: "121", and a persistent CARGO_TARGET_DIR that is not shared with the release job or with xla-compile (a link job emits different artifacts and would repeatedly invalidate a check-only cache).

The design decisions this issue must settle, rather than leaving to the implementer:

Profile is forced, not chosen. On this host the debug profile cannot link these targets at all. It fails with hundreds of relocation truncated to fit: R_AARCH64_CALL26 errors against ordinary libstd and compiler_builtins symbols, because the unoptimized binary exceeds the AArch64 direct-branch range. A link gate must therefore use --release. The cheaper debug link is not an option.

Cost is the reason this is a separate decision and not a step appended to xla-compile. Linking pulls in the whole IREE runtime and the MLX C++ library, so it is minutes rather than seconds even warm, and a cold release build of mlxcel-server with cuda,xla-iree measured 17 minutes 6 seconds on the GB10 runner. xla-compile is cheap enough to run on every Rust PR; a link job may not be, and that tradeoff is what this issue decides.

What to link. Weigh these in the issue's resolution and record the choice in the workflow file:

• cargo test --release --features cuda,xla-iree --test xla_prepared_prefill --no-run, the smallest thing that exercises the recipe. xla_prepared_prefill and molmo2_xla_vision_parity are the two [[test]] targets gated on xla-iree (Cargo.toml:327-333). --no-run links without executing, so no GPU contention with development work on the same host.
• cargo build --release --features cuda,xla-iree --bin mlxcel-server, closer to what ships but much more expensive, and the target whose 17m06s cold time is measured above.

When to run it. Running on every Rust PR is one option. The alternative is a schedule, or a narrower path filter over build.rs and scripts/iree/**, trading detection latency for runner time. A narrower trigger is already expressible: the changes job's rust filter (.github/workflows/ci.yml:77-84) was extended in #1282 to include build.rs and scripts/iree/**, so a sibling filter output keyed on just those two paths is a small addition to the existing dorny/paths-filter step.

Scope

In scope: a new job in .github/workflows/ci.yml, its trigger, its target directory, and the comment recording its cost and coverage next to the existing xla-compile exclusion list at .github/workflows/ci.yml:296-307. If a narrower path filter is chosen, a new output on the changes job.

Out of scope: running the XLA test suites (that stays excluded for GPU contention, per .github/workflows/ci.yml:297-299); clearing the dead-code backlog and widening RUSTFLAGS to -D warnings (separate issue); linking the macOS or IREE_DIST recipes (see below, no runner exists).

Implementation Notes

• Reuse: copy the provisioning and target-directory steps from xla-compile (.github/workflows/ci.yml:324-347) rather than writing a new setup path. scripts/iree/setup-cuda.sh is idempotent and reuses ~/.cache/mlxcel/iree-cuda-<version>. Keep the if: github.repository == 'lablup/mlxcel' guard so fork PRs never queue on the self-hosted runner.
• Constraints: --release is mandatory (see above). The GB10 runner is shared with development work, so the job must not execute GPU work, only link. Give it its own CARGO_TARGET_DIR under $HOME/.cargo-target/, matching the naming of mlxcel-xla-ci and mlxcel-clippy-ci.
• Coverage the job will not have: the IREE_DIST recipe (build.rs:180-220) and the macOS IREE_MACOS_HOME recipe (build.rs:38-90) are still unverified by any link, on any machine, because no runner has either distribution. PR fix(build): link libc after the IREE archives so integration tests link #1275 deliberately left both unchanged for that reason. A gate that only covers IREE_CUDA_HOME must say so in its comment rather than implying those paths are covered.
• Edge cases: a cold runner pays the one-time IREE runtime build; set timeout-minutes high enough to absorb it (xla-compile uses 120). A purged CARGO_TARGET_DIR pays the full cold link, so the timeout must accommodate the 17m06s figure plus the rest of the workspace.

Acceptance Criteria

• A job in .github/workflows/ci.yml links at least one target under --features cuda,xla-iree and fails its check run when the link fails.
• The job's trigger (every Rust PR, path-filtered, or scheduled) and its measured cost are recorded in a comment in the workflow file, adjacent to the existing xla-compile exclusion list.
• The comment states explicitly that the IREE_DIST and macOS IREE_MACOS_HOME recipes in build.rs remain unverified, so the residual gap stays recorded rather than assumed shut.
• Demonstrated by reintroducing the defect: temporarily reverting the -lc entry added in PR fix(build): link libc after the IREE archives so integration tests link #1275 (build.rs:143-176) makes the new job fail, and restoring it makes the job pass. This mirrors how PR chore(ci): compile the OpenXLA feature combinations #1282 demonstrated its own gate.
• The job does not execute GPU work, so it does not contend with development on the GB10 host.

Verification

# On the GB10 runner, with the IREE runtime provisioned:
bash scripts/iree/setup-cuda.sh
eval "$(bash scripts/iree/setup-cuda.sh --env)"
export MLX_CUDA_ARCHITECTURES=121

# The chosen link command, e.g.:
cargo test --release --features cuda,xla-iree --test xla_prepared_prefill --no-run

# Negative control: revert the `-lc` entry in build.rs and confirm the same
# command fails with the "DSO missing from command line" error above, then
# restore it.

A pass is the link command exiting 0 with the -lc entry present, and exiting non-zero with it removed. Record both outcomes in the PR.

Technical Considerations

Related: #1274 (the link failure), #1275 (the fix), #1282 (the compile gate this extends).