ci: stop running the Rust suite for documentation-only changes - #485
Conversation
`^scripts/` and `^\.github/workflows/` were accurate proxies for "paths the Rust jobs depend on" when every script and workflow in this repository served the build. ADR-0477 ended that: it added scripts/docs-publish-build.sh, scripts/docs-publish-latest-stable-version.sh, docs-publish.yml and docs-drift.yml, none of which any Rust job invokes. So a pull request touching only the documentation site and its publishing script was classified code-touching and ran the full suite — about forty minutes of compiling six e2e test binaries for a change no Rust file could observe. That is what happened to the docs-site PR this came out of. DOCS_ONLY_PATTERN removes those paths from the candidate list before the deny-list is applied. Three properties are deliberate: - The broad patterns stay broad. A script or workflow added tomorrow still defaults to code-touching; the exceptions are named individually, never by prefix, so nothing new is exempted by accident. - Each exception is a file the Rust jobs demonstrably never invoke. ci.yml runs stage-openssl-static.sh, assert-static-openssl.sh and crates/eval/scripts/test-scripts.sh, and nothing else under scripts/; docs-drift.yml and docs-publish.yml are separate workflows that ci.yml neither calls nor shares a job with. - ADR-0322 §3's fail-safe direction is unchanged. This only ever moves a named path from "code" to "docs", never the reverse, and every early return still defaults to code_changed=true. The patterns move into patterns.sh so test-patterns.sh exercises the strings CI actually uses rather than a copy that can drift. This logic has been wrong before without anything catching it — a `set -e` interaction once made every genuinely docs-only PR fail closed — and it is what stands between a code change and its test suite, so a mistake the other way is worse. Both directions are asserted, including that an unrecognised script or workflow still classifies as code. That test runs in the `changes` job, not in `test`: `test` is gated on the very output being validated, so a pattern wrong in the skip direction would take the suite down with it and go unreported. `changes` is unconditional and the check costs a second. This pull request touches .github/actions/, so it runs the full suite itself — which is correct, and the first thing the change should be judged on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P7i38GTnkhpkBbhb8XFXop
There was a problem hiding this comment.
🟡 Changes recommended
The classifier currently masks grep -Ev "$DOCS_ONLY_PATTERN" errors in a way that can fail open to “docs-only” and inadvertently skip the Rust suite.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Updates the CI “docs-only fast path” classifier to treat a small set of documentation-site publishing files as docs-only (even though they live under broad code-touching prefixes), and adds an on-PR test to prevent misclassification regressions that could accidentally skip the Rust suite.
Changes:
- Introduces shared classifier regexes (
CODE_PATTERN,DOCS_ONLY_PATTERN) and updates the composite action to pre-filter docs-only exceptions before applying the code-touching deny-list. - Adds a lightweight pattern test script and runs it in the unconditional
changesjob to validate both skip and non-skip classifications. - Documents the new exception mechanism as an ADR amendment.
File summaries
| File | Description |
|---|---|
| docs/adr/0322-ci-docs-only-fast-path.md | Adds an amendment documenting the rationale and safety properties of the new docs-only exceptions. |
| .github/workflows/ci.yml | Runs the classifier pattern test in the unconditional changes job before classification. |
| .github/actions/classify-changes/test-patterns.sh | New test harness asserting expected docs-only vs code-touching classification cases. |
| .github/actions/classify-changes/patterns.sh | New shared location for the classification regex patterns used by both CI and tests. |
| .github/actions/classify-changes/action.yml | Sources shared patterns, applies docs-only exceptions pre-filter, and updates pattern concatenation logic. |
Review details
- Files reviewed: 5/5 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| CODE_CANDIDATES=$(echo "$CHANGED_FILES" | grep -Ev "$DOCS_ONLY_PATTERN") || true | ||
|
|
There was a problem hiding this comment.
Reviewed the change to .github/actions/classify-changes (patterns.sh extraction, the new DOCS_ONLY_PATTERN, test-patterns.sh, and the ADR-0322 amendment) against the base main. The classification logic itself checks out: DOCS_ONLY_PATTERN is anchored full-path (no accidental prefix matches), applied before CODE_PATTERN in both action.yml and the test harness, the changes job runs test-patterns.sh unconditionally and ahead of the classify step (so a bad pattern fails the job and the existing fail-safe needs.changes.result != 'success' catches it), and every exception path was verified against ci.yml to confirm no Rust job actually invokes it. Test counts in the PR description match the file exactly (9/12/2 = 23).
One discrepancy: DOCS_ONLY_PATTERN also exempts scripts/generate-docs-llms-full.sh, which is neither named in the PR description's "each exception is a file the Rust jobs demonstrably never invoke" list nor recorded in the new ADR-0322 amendment text — even though it reverses that same ADR's prior (2026-08-16) amendment, which explicitly logged this script's code-classification as an accepted "Cost." The reclassification is itself correct (the script is only invoked by docs-drift.yml), so this is a documentation-completeness gap rather than a functional bug.
| # | ||
| # The fail-safe direction of ADR-0322 is unchanged: this only ever moves a named | ||
| # path from "code" to "docs", never the reverse. | ||
| DOCS_ONLY_PATTERN='^(\.github/workflows/docs-(drift|publish)\.yml|scripts/(docs-publish-build|docs-publish-latest-stable-version|generate-docs-llms-full)\.sh)$' |
There was a problem hiding this comment.
DOCS_ONLY_PATTERN exempts scripts/generate-docs-llms-full.sh in addition to the four files named in the PR description and in this ADR amendment (docs-publish-build.sh, docs-publish-latest-stable-version.sh, docs-drift.yml, docs-publish.yml). The reclassification itself is correct — ci.yml never invokes this script — but it silently reverses the prior ADR-0322 amendment (2026-08-16), which explicitly recorded this script's code-classification as an accepted 'Cost,' and neither the PR description nor the new ADR-0322 amendment text mentions the reversal. Worth naming explicitly in the ADR/PR text so the decision record stays complete, per this PR's own stated principle that exceptions are 'named individually.'
…eversal Two review findings. **The docs-only filter failed open.** `|| true` masked a grep *error* (exit above 1) as well as the ordinary "no lines selected", leaving CODE_CANDIDATES empty so the deny-list matched nothing and the pull request was classified documentation-only. A malformed pattern would have silently skipped the entire Rust suite — the precise inversion of ADR-0322 §3, in a change whose whole argument was that the fail-safe direction is preserved. Exit 1 is now distinguished from a real error, which emits code_changed=true. Verified by running the filter with an unbalanced bracket: it reports the grep failure and defaults to the full suite. **The ADR did not admit what it was reversing.** The 2026-08-16 amendment classified ^scripts/ as code to protect stage-openssl-static.sh and assert-static-openssl.sh, and explicitly accepted, as a named "Cost", that scripts/generate-docs-llms-full.sh would be swept up with them. This change reverses that specific decision, and neither the amendment nor the pull request said so. Both now do: the earlier Cost paragraph is marked superseded, and the new amendment states the reversal, why the trade changed (three scripts with one exception has become five with three), and that the protection the earlier amendment existed for is untouched — the two OpenSSL scripts are not exempted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P7i38GTnkhpkBbhb8XFXop
There was a problem hiding this comment.
Reviewed this commit, which is a follow-up fix addressing two prior review findings on PR #485: the || true that masked grep errors in the docs-only filter (now replaced with an explicit exit-status check that distinguishes "no match" from a real grep failure and fails safe to the full suite, mirroring the existing deny-list grep's error handling), and the ADR not disclosing that it reverses the 2026-08-16 amendment's accepted cost for generate-docs-llms-full.sh (now an explicit ADR-0322 amendment section that states the reversal, its rationale, and confirms the OpenSSL-script protections are untouched). Independently verified that docs-drift.yml/docs-publish.yml are not invoked by and share no jobs with ci.yml, and that test-patterns.sh sources patterns.sh directly so it can't drift from the strings CI actually uses. No new defects found.
A docs-only pull request currently runs the full Rust suite — about forty minutes compiling six e2e test binaries for a change no Rust file can observe. #481 is sitting in exactly that right now, which is what prompted this.
Why it happens
^scripts/and^\.github/workflows/were accurate proxies for "paths the Rust jobs depend on" when every script and workflow here served the build. ADR-0477 ended that, addingscripts/docs-publish-build.sh,scripts/docs-publish-latest-stable-version.sh,docs-publish.ymlanddocs-drift.yml— none of which any Rust job invokes.The change
DOCS_ONLY_PATTERNremoves those paths from the candidate list before the deny-list is applied. Three properties are deliberate:ci.ymlrunsstage-openssl-static.sh,assert-static-openssl.shandcrates/eval/scripts/test-scripts.sh, and nothing else underscripts/.docs-drift.ymlanddocs-publish.ymlare separate workflowsci.ymlneither calls nor shares a job with.code_changed=true.Tests
The patterns move into
patterns.shsotest-patterns.shexercises the strings CI actually uses rather than a copy that can drift.This logic has been wrong before with nothing catching it — a
set -einteraction once made every genuinely docs-only PR fail closed — and it is what stands between a code change and its test suite, so a mistake in the other direction is worse. Both directions are asserted, 23 cases, including that an unrecognised script or workflow still classifies as code:The test runs in the
changesjob, not intest:testis gated on the very output being validated, so a pattern wrong in the skip direction would take the suite down with it and go unreported.changesis unconditional and the check costs a second.ADR-0322 carries an amendment recording all of this.
Note
This PR touches
.github/actions/, so it runs the full suite itself. That is correct, and the first thing it should be judged on.🤖 Generated with Claude Code
https://claude.ai/code/session_01P7i38GTnkhpkBbhb8XFXop