ci: move every action onto a Node 24 runtime, and let the eval gate see its own edits - #4
Merged
Merged
Conversation
…ee its own edits
Every run in all three workflows carried:
Node.js 20 is deprecated. The following actions target Node.js 20 but
are being forced to run on Node.js 24: actions/checkout@v4,
actions/github-script@v7, actions/setup-node@v4, pnpm/action-setup@v4
The runner is already executing them on 24; the forcing is a grace
period, not a fix. 14 references across ci.yml, evals-pr.yml and
release.yml move to the current majors, each checked against how this
repo actually uses the action rather than bumped on faith:
| action | v4 → | breaking change | why it does not bite here |
|---|---|---|---|
| checkout | v7 | blocks fork-PR checkout for `pull_request_target` / `workflow_run` | neither event is used |
| setup-node | v7 | v5 auto-caches on `packageManager`, v6 narrows that to npm, v7 drops the dummy `NODE_AUTH_TOKEN` | `cache: pnpm` is explicit in all four steps, so auto-cache never applies |
| github-script | v9 | `require('@actions/github')` gone, `getOctokit` now an injected parameter | scripts `require` only `fs` and `path`, and never name `getOctokit` |
| pnpm/action-setup | v6 | none — input surface identical to v4 | no inputs are passed |
setup-node v7 is an improvement rather than a risk for `release.yml`.
Publishing there is npm Trusted Publishing over OIDC with no
`NODE_AUTH_TOKEN` anywhere in the repo, and the dummy value v4 exported
is what actions/setup-node#1558 removed precisely because it "could
corrupt the user's .npmrc ... causing confusing behavior during OIDC
publish".
Also: `evals-pr.yml` now lists itself in both paths filters. Editing
the gate did not run the gate, so a reordered step, a bumped action or
a new trigger shipped without the workflow it edits ever executing —
the same shape as the three vacuous passes 7033f57 and dbf0bd6 fixed.
That is not a hypothetical here: without it this very commit would put
`github-script@v9` into the eval workflow with nothing exercising it,
since `.github/**` matches none of the detector, scoring, language, CLI
or evals paths.
`release.yml` stays unexercised — it triggers on `release: published`
and cannot run until the next release is cut. Its four actions are the
same versions the other two workflows exercise on this PR.
`pnpm verify` (2407 tests) green. No product code touched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Eval replay diffReplayed at: 2026-08-26T09:56:23.369Z
No regressions outside ±10% tolerance — eval signal is stable. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every run in all three workflows carried this annotation:
The runner is already executing them on 24 — the forcing is a grace period, not a fix. 14 references move to current majors, each checked against how this repo actually uses the action rather than bumped on faith:
actions/checkoutpull_request_target/workflow_runactions/setup-nodepackageManager; v6 narrows that to npm; v7 drops the dummyNODE_AUTH_TOKENcache: pnpmis explicit in all four steps, so auto-cache never appliesactions/github-scriptrequire('@actions/github')gone;getOctokitnow an injected parameterrequireonlyfsandpath, and never namegetOctokitpnpm/action-setupThe one that needed real checking
release.ymlpublishes to npm via Trusted Publishing over OIDC, and setup-node v7 removes aNODE_AUTH_TOKENexport. That reads like a publish-breaking change, so I checked it rather than assuming: there is noNODE_AUTH_TOKENanywhere in the repo, and actions/setup-node#1558 removed the dummy value precisely because it "could corrupt the user's .npmrc ... causing confusing behavior during OIDC publish". For this repo v7 is an improvement, not a risk.Letting the gate see its own edits
evals-pr.ymlnow lists itself in both paths filters. Editing the gate did not run the gate — a reordered step, a bumped action or a new trigger shipped without the workflow it edits ever executing. Same shape as the vacuous passes in7033f57anddbf0bd6.Not hypothetical: without it, this commit would put
github-script@v9into the eval workflow with nothing exercising it, since.github/**matches none of the detector / scoring / language / CLI / evals paths. With it, this PR runs that workflow and its comment step.What this PR does and does not verify
ci.yml(build, typecheck, test, publish smoke) andevals-pr.yml(replay, diff, and thegithub-scriptcomment step) on all four new action versions.release.ymltriggers onrelease: publishedand cannot run until the next release is cut. Its four actions are the same versions the other two workflows exercise on this PR.pnpm verify(2407 tests) green locally. No product code touched — no findings, no scores, no version bump.🤖 Generated with Claude Code