ci(nix): key the Bazel-Dev lane to the hermetic toolchain identity so it can hit the warm cache

[b7r6]

What and why

The common case first. The vast majority of PRs change a small dependency cone (a handful of crates). In that regime, with a warm remote cache, the Bazel lanes are fast because their actions are cacheable — but the Nix / Bazel Dev lane can't participate: it lands 0 remote cache hits of its 2723 actions (2723 processes: 1142 internal, 1700 processwrapper-sandbox — every compile runs locally), so it pays a full from-source build (~12m) on every PR regardless of cone size. By contrast the Bazel 9 Native lane, against the same cache endpoint, gets 1877 remote cache hits + 803 disk cache hits of 3915 actions (3915 processes: 803 disk cache hit, 1877 remote cache hit, 1351 internal, 3 processwrapper-sandbox). This change targets the cache-key mismatch that keeps the Bazel Dev lane cold so it can read the same warm cache.

Why the lane misses on keys: it compiles inside nix develop, where the default C++/Rust toolchain resolves to nix-store paths (/nix/store/<hash>-clang|gcc|rust) that are baked into every action key — so its digests cannot match a cache warmed off a non-nix toolchain. The fix keys the lane on a stable, relocatable toolchain identity instead of host paths. (On runs where the lane also hits Remote Cache: UNAVAILABLE transient connectivity, hits are zero for that separate reason; the key mismatch is the standing cause this change removes, and it is the one demonstrated by the aquery proof below.)

The shape of the problem (measured fanout curve)

No-op comment at increasing dependency-cone depth, same read-only staging cache. Every cell is a clickable run. This is the shape that makes the case:

Cone (crates)	Bazel 9 Native (cacheable)	Cargo Ubuntu (uncacheable)	Cargo Windows (uncacheable)	Bazel Dev (0-hit, today)
T1 leaf — 1	3m02s	5m35s	11m55s	12m17s
T2 worker — 2	2m58s	6m17s	14m16s	12m19s
T3 store — 5	4m09s	7m39s	14m28s	12m49s
T4 error — 9	5m51s	6m14s	11m49s	12m30s

Runs: T1 bazel / cargo · T2 bazel / cargo · T3 bazel / cargo · T4 bazel / cargo

Reading it:

• Bazel 9 Native scales with the cone — ~3m for a small-cone PR, climbing with cone size. That is the cache working: small cone, little novel work, fast.
• Cargo is flat and high at every cone — there is no fine-grained cross-run action cache for cargo on any platform, so even a 1-crate leaf change pays the full ~12m Windows / ~6m Ubuntu tax. Cargo cannot play this game.
• Bazel Dev is flat ~12m at every cone — not because its work is large, but because it hits the cache 0 times (the bug here). Fixed, it should track the Bazel 9 curve (~3m on the common small-cone PR), not sit at a flat 12m.

(Spread check: the T1 leaf Bazel 9 lane measured 3m02s and 3m25s on two independent runs — tight enough that the curve's shape is not noise.)

So for the common case — warm cache, small-cone PR — this change moves the Bazel Dev lane from a flat ~12m into the fast cacheable regime.

The change

Two files, minimal and self-contained:

• ci.bazelrc: factor the hermetic LLVM platform + downloaded Rust toolchains that nl-rbe already uses into a new, endpoint-free nl-toolchain config; nl-rbe inherits it via --config=nl-toolchain (no behavior change for nl-rbe — same flags, now shared). These toolchains key on downloaded content, not host paths, so they produce identical action digests on a bare runner, inside nix develop, or on a remote worker.
• .github/workflows/nix.yaml: apply --config=nl-toolchain to the Bazel Dev lane. Cache-read only — no remote executor is added; the lane keeps running locally, it just keys its compiles against the hermetic identity so they can hit a warm cache.

How was this verified?

• DEMONSTRATED (measured, linkable): the toolchain path signatures align. A bazel aquery proof compares the compile action command lines three ways — (A) inside nix develop with the default toolchain, (B) inside nix develop with --config=nl-toolchain, (C) on a bare runner with --config=nl-toolchain — and shows B matches C (the warmer identity) and no longer references /nix/store toolchains, whereas A does: https://github.com/b7r6/nativelink/actions/runs/37059855343 . That path-signature alignment is the precondition for a warm read. (The proof prints the signatures/counts for inspection rather than hard-failing the job on mismatch; the comparison is in its uploaded aquery-* artifacts.)

• Green + the key-match is live, not just theoretical (measured, linkable): on the fork verification run the Bazel Dev lane builds successfully with --config=nl-toolchain and its remote-cache hit count moves from 0 to 18 (3915 processes: 18 remote cache hit, 1351 internal, 2665 processwrapper-sandbox) — Bazel Dev job, full Nix run, all lanes green. The 18 hits are whatever already sits in the staging cache under the hermetic identity; it is small precisely because nothing has warmed the Bazel-Dev cone under this key yet (see the dependency below). That the number is nonzero at all is the direct, non-aquery confirmation that the lane now keys against the warm identity.

• PROJECTED (NOT measured end-to-end), with its basis stated: once a writer warms this key, the lane should drop from ~12m (the flat curve above) toward the regime the Native Bazel 9 lane already demonstrates against the same cache endpoint — that lane hits 2680/3915 actions cached (1877 remote + 803 disk, 68%) and finishes in ~5-6m. The projection is "the Bazel Dev lane, once keyed identically, reaches a comparable hit rate," i.e. ~12m → ~4-6m (design target ~250s). It is a projection, not a measurement: see the caveat.

  Why it is not measured here (stated plainly): realizing the warm read needs a writer that populates this key, and I verified this fork has no such writer — it carries 0 Actions secrets / 0 variables, so its CI reads the staging cache anonymously and cannot write Bazel-Dev-keyed (nl-toolchain) artifacts into it, and its main-push warmer is non-functional for the same reason. I could not, at any budget, populate a warm nl-toolchain-keyed cache from this fork. So this PR presents the demonstrated key-match plus the projection anchored to the Native lane's measured 68% warm hit rate, rather than a fork-measured warm number I cannot honestly produce.

The one honest caveat (worst case, acknowledged)

The warm hit is not yet demonstrable on this fork, and realizing it needs one upstream line: the trusted main-push cache warmer must also build with --config=nl-toolchain, so it warms the same key this lane now requests. Today's Native warmer uses bare --extra_toolchains without the @llvm platform pin, so it warms a different key. Until that one-line change lands, this lane's hit rate is unchanged — it does not get slower, it just doesn't yet get faster. I'm happy to send that warmer one-liner as a follow-up the moment this lands, or fold it in here if you'd prefer it in one PR. (The fork's own main-push warmer is broken for unrelated reasons, which is why the end-to-end warm number is projected rather than measured here.)

Risk

Low. Cache-read keying change only — no remote executor, no cache-write change, no ci-mode change. The nl-toolchain extraction is a pure refactor of flags nl-rbe already carried. Worst case if the warmer line never lands: the Bazel Dev lane's hit rate is unchanged (never slower). macOS lanes do not apply this config (LRE auto-applies conflicting platforms there, already guarded by the setup action).

AI assistance

An agent (Claude Opus 4.8) drafted the config refactor, ran the aquery proof, the fanout-curve measurement, and the fork verification, and wrote this description; a human (b7r6) directed it, reviewed every line, and is answerable for all of it. The commit carries a Co-Authored-By trailer.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
nativelink	Ready	Preview	Oct 3, 2026 4:40am UTC
nativelink-aidm	Ready	Preview	Oct 3, 2026 4:40am UTC

[MarcusSorealheis]

LGTM