Repository navigation
ci(l1,l2): key the Rust cache by runner image so pools cannot span glibc versions - #7154
Conversation
…versions
`Ethrex Release` has been failing on main since 2026-08-12 in
`build-ethrex (ubuntu-22.04, l2)`, with `parity-scale-codec` dying on eighteen
unresolved imports of its own re-exports. The dependency is not at fault; the
first error in the log is:
libparity_scale_codec_derive-*.so: /lib/x86_64-linux-gnu/libc.so.6:
version `GLIBC_2.39' not found
rust-cache keys on `runner.os` and `runner.arch`, which are `Linux` and `x64`
for both ubuntu-22.04 and ubuntu-24.04, so the restored key was
`v0-rust-l2-Linux-x64-...` for both. The `l2` pool is written by
`pr-main_l2.yaml:integration-test` on `ubuntu-latest`, now ubuntu-24.04 with
glibc 2.39, and read by `build-ethrex` on ubuntu-22.04 with glibc 2.35. The
restored proc-macro `.so` cannot be dlopened there, so `extern crate
parity_scale_codec_derive` fails and every `crate::Decode`/`Encode`/`Error`
re-export collapses. The failure names the dependency rather than the cache,
which is what made it look like a dependency problem.
Fold `ImageOS` into the cache key. Image family rather than `ImageVersion`,
which changes with every runner image release and would evict the pools weekly;
family is exactly the granularity that separates glibc floors. It is read
through `$GITHUB_OUTPUT` because the `env` context is not reliably available to
`with:` on a `uses:` step inside a composite action.
This also closes the same hazard latent in the `l1` pool, whose writer
(`pr-main_l1.yaml:test`) is ubuntu-22.04 while several readers are
ubuntu-latest. That direction happens to be forward-compatible, so it has not
broken yet, but it would the moment the writer matrix moves to 24.04.
The key change invalidates every existing entry once; the pools refill on the
next main build.
ilitteri
added
L2
|
3c3b6f4af (devnet-7) |
this client (devnet-8) | |
|---|---|---|
COLD_STORAGE_ACCESS |
3000 | 2100 |
ACCOUNT_WRITE |
8000 | 9000 |
CALL_VALUE |
10300 | 11300 |
| access-list storage key | full cold cost | cold − WARM_ACCESS |
Five of the 45 failed even while the client was still on devnet-7, so those are
skew against 3c3b6f4af itself; the other 40 appeared with the devnet-8 move.
The two groups are commented separately in EXTRA_SKIPS because they lift at
different times.
Corroborating that these are spec-base skew and not client bugs: ethrex passes
the five original cases against the tests-zkevm@v0.6.2 versions of the same
tests, and the eth-act witness dashboard has ethrex at 22829/22829 on
eels/consume-engine-witness and 19413/19413 on stateless-validator sp1 for
that bundle. Diffing witness_codes_reset_delegation across the two fixture
sets: pre and genesisBlockHeader are byte-identical and it is the same
transaction hash, yet postState, receipts, stateRoot, receiptTrie and
blockAccessListHash all differ — the receipt's cumulativeGasUsed goes
30816 → 24653. Same input, different output.
The regular vectors/ bundle has no eip8025_optional_proofs tests, so
make test-levm never sees any of this.
The pin cannot move today. eip8025_optional_proofs does not exist on the
current forks/amsterdam default branch, and 3c3b6f4af has diverged from it
(124 ahead, 315 behind), so it was never merged there. devnets/glamsterdam/8
is 315 commits behind 3c3b6f4af: it has the devnet-8 gas schedule but not the
#3278 schema, so vectors filled from it would carry the pre-#3278 69-byte
statelessOutputBytes this client cannot parse. No upstream revision has both.
Removal: bump SPEC_SHA in scripts/gen_stateless_vectors.sh once an
upstream revision carries both eip8025_optional_proofs and the devnet-8 gas
schedule, then delete the 40-entry group. Re-check the original five at the same
time; if they still fail against a matching base they are real execution bugs
and must not stay skipped.
Why this file previously claimed an Amsterdam-wide skip
parse_and_execute used to drop every network >= Fork::Amsterdam fixture from
the stateless run. Every test in the tests-zkevm bundle it read is
network: Amsterdam (23,946 of 23,946), so the stateless suite executed nothing
at all while reporting success. parse_and_execute now fails any stateless
fixture file that runs zero tests without a named skip, so a structural skip
cannot silently empty the suite again.
ZisK guest program hash changes with the unsync_cell gate
Where: crates/common/types/block.rs, transaction.rs.
The gate on the single-threaded unsync_cell::OnceCell moved from
all(feature = "eip-8025", target_arch = "riscv64") to
all(feature = "zisk", target_arch = "riscv64") when the eip-8025 feature was removed.
The guest ELFs were previously built --features "<zkvm>-build-elf,ci", which never enabled
eip-8025, so they compiled the atomic once_cell variant. bin/zisk/Cargo.toml does enable
ethrex-common/zisk, so the ZisK guest now compiles the unsafe impl Sync cell instead.
That changes the ELF bytes and therefore the program hash and verification key.
This is intended (the guest is single-threaded, so the unsync cell is sound and cheaper), but it
is a VK change rather than a no-op refactor, and the diffstat presents it as a file rename
(eip8025_cell.rs → unsync_cell.rs). Anyone pinning a ZisK VK across this change must
re-register it. The stateless-validator crate now forwards ethrex-common/zisk from its own
zisk feature so the two ZisK guests do not disagree on the cell type.
Release signing key is an unprotected repository secret
Where: .github/workflows/tag_release.yaml.
MINISIGN_SECRET_KEY is a plain repository secret. There is no environment: on
finalize-release or dry-run-release-assets, and gh api repos/lambdaclass/ethrex/rulesets
shows only branch-targeted rulesets, so the github.ref_type == 'tag' condition is a workflow
check rather than an enforced boundary: anyone who can push a tag can reach the signing key.
This is a repository-settings change, not a code change, so it is recorded here rather than
fixed in the tree. Recommended:
- Move
MINISIGN_SECRET_KEY/MINISIGN_PASSWORDinto a GitHub Environment with required
reviewers, and addenvironment:to the two jobs that sign. - Add a ruleset targeting
refs/tags/v*restricting who may create release tags.
Until then, the compromise of that key is silent and durable: signatures would still verify
against the committed .github/minisign.pub.
🤖 Kimi Code ReviewThis change correctly fixes a glibc version mismatch issue in CI caching. The logic is sound and the implementation follows GitHub Actions best practices. Key points:
Minor note: The extensive comment (lines 40-55) is justified given the subtlety of the failure mode—future maintainers need to understand why this discriminator exists to avoid regressions if they refactor the caching strategy. No changes required; this is a clean fix. Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt |
🤖 Codex Code ReviewNo findings. The change in action.yml looks correct: it adds the runner image family as an extra Only minor caveat: action.yml still collapses self-hosted runners into Automated review by OpenAI Codex · gpt-5.4 · custom prompt |
🤖 Claude Code ReviewReview: PR 7154 — Key the Rust cache by runner image familyThis is a CI-only change to CorrectnessThe root-cause analysis and fix are sound and I verified the key claims against the current repo state:
SecurityNo concerns. Minor observations (non-blocking)
SummaryWell-scoped, correctly targeted fix with solid root-cause analysis backed by verifiable evidence from the workflow files (writer/reader OS mismatch on the Automated review by Claude (Anthropic) · sonnet · custom prompt |
`Report PR Line Changes` failed on this branch alone with a runner-internal "Index was out of range" during post-job cleanup. That job checks out both the PR head and main in one job and runs this composite action from each, so a step list that differs between the two revisions desynchronises the runner's post-step bookkeeping — and this is the PR that changes the step list. Folding the `ImageOS` read into the existing `rustver` step keeps the action at three steps, matching main, so the two checkouts agree. The cache key is unchanged in content; only where the value is computed moves.
…lambdaclass#7166) **Motivation** `Ethrex Release` is still red on `main` after lambdaclass#7154, failing in `build-ethrex (ubuntu-22.04, l2)` with the same error that PR set out to fix ([run 32302415108](https://github.com/lambdaclass/ethrex/actions/runs/32302415108/job/96227727106), on `7277c63ee`, which already contains it): ``` error: …/libparity_scale_codec_derive-….so: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.39' not found ``` The diagnosis in lambdaclass#7154 holds: a proc-macro `.so` built on ubuntu-24.04 is restored into an ubuntu-22.04 job, `rustc` cannot dlopen it, and `parity-scale-codec` collapses into eighteen `E0432`s that name the dependency rather than the cache. The fix is what did not work. lambdaclass#7154 passed the image family to `Swatinem/rust-cache` through its `key` input. `key` is only honoured on the branch that builds a per-job key: ```ts // rust-cache src/config.ts const sharedKey = core.getInput("shared-key"); if (sharedKey) { key += `-${sharedKey}`; } else { const inputKey = core.getInput("key"); if (inputKey) key += `-${inputKey}`; // …job id… } ``` Every job that sets `cache-pool` sets `shared-key`, so it takes the first branch and `key` is dropped. The failing run shows the input arriving and the key ignoring it: ``` with: shared-key: l2 key: ubuntu22 … Runner image family: ubuntu22 Restore Key: v0-rust-l2-Linux-x64-73d6c353 Cache Key: v0-rust-l2-Linux-x64-73d6c353-18b5747f Restored from cache key "v0-rust-l2-Linux-x64-73d6c353-18b5747f" full match: true. ``` Same pool, same key, both images. The 24.04 writer (`pr-main_l2.yaml:integration-test`, `runs-on: ubuntu-latest`) and the 22.04 reader (`tag_release.yaml:build-ethrex`) still share one entry. **Description** Fold the image family into the shared key itself: `shared-key: <pool>-<image family>`, so it lands on the branch rust-cache actually uses. | | before | after | | --- | --- | --- | | pooled job on 22.04 | `v0-rust-l2-Linux-x64-…` | `v0-rust-l2-ubuntu22-Linux-x64-…` | | pooled job on 24.04 | `v0-rust-l2-Linux-x64-…` | `v0-rust-l2-ubuntu24-Linux-x64-…` | The shared key is left empty when `cache-pool` is unset, which keeps those jobs (e.g. `pr_nostd.yaml`) on rust-cache's per-job key. `key: ${{ image_family }}` stays on the step for exactly that case, where it is honoured. `inputs.cache-pool` reaches the script through `env:` rather than direct `${{ }}` interpolation. **Consequences** `l2` has no 22.04 writer: the pool is written by `integration-test` on `ubuntu-latest`. So `build-ethrex (ubuntu-22.04, l2)` gets a cold build rather than a poisoned warm one. That is the correct outcome and the only one available without a 22.04 writer for that pool; giving the release job its own writable pool would warm it, but each entry runs 1.2-2.75GB against a 10GB LRU budget, so it is a separate call and left out here. As in lambdaclass#7154, changing the key invalidates existing entries once. Pools refill on the next `main` build. **How to test** `tag_release.yaml` has no `pull_request` trigger, so the real check is that `build-ethrex (ubuntu-22.04, l2)` goes green on the next push to `main`. Before then, any job's log shows the key now carrying the image family: ``` v0-rust-l2-ubuntu22-Linux-x64-… ``` **Checklist** - [ ] Updated `STORE_SCHEMA_VERSION` (crates/storage/lib.rs) if the PR includes breaking changes to the `Store` requiring a re-sync.
2 participants
Motivation
Ethrex Releasehas been red onmainsince 2026-08-12, failing inbuild-ethrex (ubuntu-22.04, l2)whileparity-scale-codecreports eighteen unresolved imports of its own re-exports. The dependency is not at fault — the first error in the log is:The proc-macro
.socame out of the Actions cache, built on a runner with a newer glibc than the one restoring it.rustccannot dlopen it, soextern crate parity_scale_codec_derivefails and everycrate::Decode/Encode/Errorre-export collapses behind it. The wall ofE0432s names the dependency rather than the cache, which is what made this look like a dependency-resolution problem.This is not caused by any recent PR. The commit immediately before the most recent merge,
06773cee4, fails identically, andtag_release.yamlonly triggers on push-to-main and tags — neverpull_request— so no PR can catch it.Description
Swatinem/rust-cachecomposes its key fromrunner.osandrunner.arch, which areLinuxandx64for both ubuntu-22.04 and ubuntu-24.04. The observed restore was:Nothing in that key distinguishes the image, so a pool shared by jobs on different images resolves to one entry:
save-cache)l2pr-main_l2.yaml:integration-testubuntu-latest→ 24.04, glibc 2.39tag_release.yaml:build-ethrexon 22.04, glibc 2.35l1pr-main_l1.yaml:testlint,daily_hive_report,daily_loc_report,deep_reorg,pr_lint_readme,pr_loc,pr_perf_levmonubuntu-latestl2is the one failing: newer glibc into older runner.l1has the same flaw pointing the other way, which is forward-compatible and so has not broken — but it would the moment that writer matrix moves to 24.04.The fix folds
ImageOSinto the cache key. Image family (ubuntu22/ubuntu24/macos15) rather thanImageVersion, which changes with every runner image release and would evict the pools weekly; family is exactly the granularity that separates glibc floors. It is read via$GITHUB_OUTPUTrather thanenv.ImageOSdirectly, because theenvcontext is not reliably available towith:on auses:step inside a composite action.Fixing it in the shared action rather than at the call site means no pool can ever span images, including pools added later.
Note on rollout
Changing the key invalidates every existing entry once. Pools refill on the next
mainbuild; the first run after merge is a cold build.How to test
The failure only reproduces on
main(the workflow has nopull_requesttrigger), so the real check is thatbuild-ethrex (ubuntu-22.04, l2)goes green on the next push tomain. Before then, the key change is visible in any job's log — the restore line should carry the image family:Checklist
STORE_SCHEMA_VERSION(crates/storage/lib.rs) if the PR includes breaking changes to theStorerequiring a re-sync.