Skip to content

ci: stop running the Rust suite for documentation-only changes - #485

Merged
verveguy merged 2 commits into
mainfrom
ci/docs-only-classification
Aug 24, 2026
Merged

ci: stop running the Rust suite for documentation-only changes#485
verveguy merged 2 commits into
mainfrom
ci/docs-only-classification

Conversation

@verveguy

Copy link
Copy Markdown
Owner

A docs-only pull request currently runs the full Rust suite — about forty minutes compiling six e2e test binaries for a change no Rust file can observe. #481 is sitting in exactly that right now, which is what prompted this.

Why it happens

^scripts/ and ^\.github/workflows/ were accurate proxies for "paths the Rust jobs depend on" when every script and workflow here served the build. ADR-0477 ended that, adding scripts/docs-publish-build.sh, scripts/docs-publish-latest-stable-version.sh, docs-publish.yml and docs-drift.yml — none of which any Rust job invokes.

The change

DOCS_ONLY_PATTERN removes those paths from the candidate list before the deny-list is applied. Three properties are deliberate:

  • The broad patterns stay broad. A script or workflow added tomorrow still defaults to code-touching. Exceptions are named individually, never by prefix, so nothing new is exempted by accident.
  • Each exception is a file the Rust jobs demonstrably never invoke. ci.yml runs stage-openssl-static.sh, assert-static-openssl.sh and crates/eval/scripts/test-scripts.sh, and nothing else under scripts/. docs-drift.yml and docs-publish.yml are separate workflows ci.yml neither calls nor shares a job with.
  • ADR-0322 §3's fail-safe direction is unchanged. This only ever moves a named path from "code" to "docs", never the reverse; every early return still defaults to code_changed=true.

Tests

The patterns move into patterns.sh so test-patterns.sh exercises the strings CI actually uses rather than a copy that can drift.

This logic has been wrong before with nothing catching it — a set -e interaction once made every genuinely docs-only PR fail closed — and it is what stands between a code change and its test suite, so a mistake in the other direction is worse. Both directions are asserted, 23 cases, including that an unrecognised script or workflow still classifies as code:

docs-only changes must skip the suite:      9 ok
code changes must run the suite:           12 ok
new paths default to code, never to docs:   2 ok

The test runs in the changes job, not in test: test is gated on the very output being validated, so a pattern wrong in the skip direction would take the suite down with it and go unreported. changes is unconditional and the check costs a second.

ADR-0322 carries an amendment recording all of this.

Note

This PR touches .github/actions/, so it runs the full suite itself. That is correct, and the first thing it should be judged on.

🤖 Generated with Claude Code

https://claude.ai/code/session_01P7i38GTnkhpkBbhb8XFXop

`^scripts/` and `^\.github/workflows/` were accurate proxies for "paths the
Rust jobs depend on" when every script and workflow in this repository served
the build. ADR-0477 ended that: it added scripts/docs-publish-build.sh,
scripts/docs-publish-latest-stable-version.sh, docs-publish.yml and
docs-drift.yml, none of which any Rust job invokes.

So a pull request touching only the documentation site and its publishing
script was classified code-touching and ran the full suite — about forty
minutes of compiling six e2e test binaries for a change no Rust file could
observe. That is what happened to the docs-site PR this came out of.

DOCS_ONLY_PATTERN removes those paths from the candidate list before the
deny-list is applied. Three properties are deliberate:

- The broad patterns stay broad. A script or workflow added tomorrow still
  defaults to code-touching; the exceptions are named individually, never by
  prefix, so nothing new is exempted by accident.
- Each exception is a file the Rust jobs demonstrably never invoke. ci.yml runs
  stage-openssl-static.sh, assert-static-openssl.sh and
  crates/eval/scripts/test-scripts.sh, and nothing else under scripts/;
  docs-drift.yml and docs-publish.yml are separate workflows that ci.yml
  neither calls nor shares a job with.
- ADR-0322 §3's fail-safe direction is unchanged. This only ever moves a named
  path from "code" to "docs", never the reverse, and every early return still
  defaults to code_changed=true.

The patterns move into patterns.sh so test-patterns.sh exercises the strings CI
actually uses rather than a copy that can drift. This logic has been wrong
before without anything catching it — a `set -e` interaction once made every
genuinely docs-only PR fail closed — and it is what stands between a code
change and its test suite, so a mistake the other way is worse. Both directions
are asserted, including that an unrecognised script or workflow still
classifies as code.

That test runs in the `changes` job, not in `test`: `test` is gated on the very
output being validated, so a pattern wrong in the skip direction would take the
suite down with it and go unreported. `changes` is unconditional and the check
costs a second.

This pull request touches .github/actions/, so it runs the full suite itself —
which is correct, and the first thing the change should be judged on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P7i38GTnkhpkBbhb8XFXop
Copilot AI lite review requested due to automatic review settings August 24, 2026 01:02

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The classifier currently masks grep -Ev "$DOCS_ONLY_PATTERN" errors in a way that can fail open to “docs-only” and inadvertently skip the Rust suite.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Updates the CI “docs-only fast path” classifier to treat a small set of documentation-site publishing files as docs-only (even though they live under broad code-touching prefixes), and adds an on-PR test to prevent misclassification regressions that could accidentally skip the Rust suite.

Changes:

  • Introduces shared classifier regexes (CODE_PATTERN, DOCS_ONLY_PATTERN) and updates the composite action to pre-filter docs-only exceptions before applying the code-touching deny-list.
  • Adds a lightweight pattern test script and runs it in the unconditional changes job to validate both skip and non-skip classifications.
  • Documents the new exception mechanism as an ADR amendment.
File summaries
File Description
docs/adr/0322-ci-docs-only-fast-path.md Adds an amendment documenting the rationale and safety properties of the new docs-only exceptions.
.github/workflows/ci.yml Runs the classifier pattern test in the unconditional changes job before classification.
.github/actions/classify-changes/test-patterns.sh New test harness asserting expected docs-only vs code-touching classification cases.
.github/actions/classify-changes/patterns.sh New shared location for the classification regex patterns used by both CI and tests.
.github/actions/classify-changes/action.yml Sources shared patterns, applies docs-only exceptions pre-filter, and updates pattern concatenation logic.
Review details
  • Files reviewed: 5/5 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines 135 to 136
CODE_CANDIDATES=$(echo "$CHANGED_FILES" | grep -Ev "$DOCS_ONLY_PATTERN") || true

@handarbeit-pruefer handarbeit-pruefer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the change to .github/actions/classify-changes (patterns.sh extraction, the new DOCS_ONLY_PATTERN, test-patterns.sh, and the ADR-0322 amendment) against the base main. The classification logic itself checks out: DOCS_ONLY_PATTERN is anchored full-path (no accidental prefix matches), applied before CODE_PATTERN in both action.yml and the test harness, the changes job runs test-patterns.sh unconditionally and ahead of the classify step (so a bad pattern fails the job and the existing fail-safe needs.changes.result != 'success' catches it), and every exception path was verified against ci.yml to confirm no Rust job actually invokes it. Test counts in the PR description match the file exactly (9/12/2 = 23).

One discrepancy: DOCS_ONLY_PATTERN also exempts scripts/generate-docs-llms-full.sh, which is neither named in the PR description's "each exception is a file the Rust jobs demonstrably never invoke" list nor recorded in the new ADR-0322 amendment text — even though it reverses that same ADR's prior (2026-08-16) amendment, which explicitly logged this script's code-classification as an accepted "Cost." The reclassification is itself correct (the script is only invoked by docs-drift.yml), so this is a documentation-completeness gap rather than a functional bug.

#
# The fail-safe direction of ADR-0322 is unchanged: this only ever moves a named
# path from "code" to "docs", never the reverse.
DOCS_ONLY_PATTERN='^(\.github/workflows/docs-(drift|publish)\.yml|scripts/(docs-publish-build|docs-publish-latest-stable-version|generate-docs-llms-full)\.sh)$'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DOCS_ONLY_PATTERN exempts scripts/generate-docs-llms-full.sh in addition to the four files named in the PR description and in this ADR amendment (docs-publish-build.sh, docs-publish-latest-stable-version.sh, docs-drift.yml, docs-publish.yml). The reclassification itself is correct — ci.yml never invokes this script — but it silently reverses the prior ADR-0322 amendment (2026-08-16), which explicitly recorded this script's code-classification as an accepted 'Cost,' and neither the PR description nor the new ADR-0322 amendment text mentions the reversal. Worth naming explicitly in the ADR/PR text so the decision record stays complete, per this PR's own stated principle that exceptions are 'named individually.'

…eversal

Two review findings.

**The docs-only filter failed open.** `|| true` masked a grep *error* (exit
above 1) as well as the ordinary "no lines selected", leaving CODE_CANDIDATES
empty so the deny-list matched nothing and the pull request was classified
documentation-only. A malformed pattern would have silently skipped the entire
Rust suite — the precise inversion of ADR-0322 §3, in a change whose whole
argument was that the fail-safe direction is preserved. Exit 1 is now
distinguished from a real error, which emits code_changed=true. Verified by
running the filter with an unbalanced bracket: it reports the grep failure and
defaults to the full suite.

**The ADR did not admit what it was reversing.** The 2026-08-16 amendment
classified ^scripts/ as code to protect stage-openssl-static.sh and
assert-static-openssl.sh, and explicitly accepted, as a named "Cost", that
scripts/generate-docs-llms-full.sh would be swept up with them. This change
reverses that specific decision, and neither the amendment nor the pull request
said so. Both now do: the earlier Cost paragraph is marked superseded, and the
new amendment states the reversal, why the trade changed (three scripts with
one exception has become five with three), and that the protection the earlier
amendment existed for is untouched — the two OpenSSL scripts are not exempted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P7i38GTnkhpkBbhb8XFXop
@verveguy
verveguy enabled auto-merge August 24, 2026 01:17

@handarbeit-pruefer handarbeit-pruefer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed this commit, which is a follow-up fix addressing two prior review findings on PR #485: the || true that masked grep errors in the docs-only filter (now replaced with an explicit exit-status check that distinguishes "no match" from a real grep failure and fails safe to the full suite, mirroring the existing deny-list grep's error handling), and the ADR not disclosing that it reverses the 2026-08-16 amendment's accepted cost for generate-docs-llms-full.sh (now an explicit ADR-0322 amendment section that states the reversal, its rationale, and confirms the OpenSSL-script protections are untouched). Independently verified that docs-drift.yml/docs-publish.yml are not invoked by and share no jobs with ci.yml, and that test-patterns.sh sources patterns.sh directly so it can't drift from the strings CI actually uses. No new defects found.

@verveguy
verveguy merged commit 9aad23c into main Aug 24, 2026
18 checks passed
@verveguy
verveguy deleted the ci/docs-only-classification branch August 24, 2026 01:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants