Skip to content

fix(ci): gate the frontend npm audit on one report, retry it, and report 'could not check' apart from advisories (#16337) - #16357

Merged
mrveiss merged 3 commits into
Dev_new_guifrom
issue-16337
Sep 12, 2026
Merged

mrveiss merged 3 commits into
Dev_new_guifrom
issue-16337

Conversation

@mrveiss

@mrveiss mrveiss commented Sep 11, 2026 •

Copy link
Copy Markdown
Owner

Closes #16337

Single-issue rationale: this changes only the Security Scan's audit gate (one workflow step, its script and its path filter). The related issues change what the gate covers or what triggers it, which is a different risk: #16131 (audit the other 12 package.json files) and #16260 (the terminal plugin push path). They ride separately.

Thinking Path

  • On 2026-09-11 merge train chore(ci): merge train 2026-09-11-c, test vehicle, do not merge (#15937) #16321 went red on Frontend Testing Suite / Security Scan with no advisory at all: npm's audit service returned a 400. The gate couldn't tell "we couldn't check" apart from "we found a high advisory".
  • The design comment on ci(security): the npm audit gate fails an audit-endpoint error exactly like a high advisory — retry, and report 'could not check' distinctly #16337 traced the cause to the step calling npm's audit service twice:
    • npm audit --json > audit-results.json || true fetched the report. That call succeeded.
    • npm audit --audit-level=high re-fetched the same data as the gate, and that second call was the one that got the 400.
  • The 400 came from npm's own fallback. Node 22 bundles npm 10.9. In npm v10.9.3, workspaces/arborist/lib/audit-report.js:323-336 tries /-/npm/v1/security/advisories/bulk and, when that fails, falls back to /-/npm/v1/security/audits/quick. That quick endpoint answered "this endpoint is being retired".
  • npm 11 has no fallback. The same file at v11.0.0 and at v11.19.1, the latest stable release (published 2026-08-26), contains no audits/quick reference at all. npm 11 supports Node ^20.17.0 || >=22.9.0.
  • So the gate should make one call, with npm 11, decide from that one report, and name the three results separately.

What Changed

Commits ed88877, 9174a08 and d5f1bfa. The third adds the review's test gaps: real OSErrors on the report and summary writes leave the exit code equal to the verdict's; the crash test asserts the traceback reaches stderr; a real subprocess emitting a non-UTF-8 byte next to an audits/quick line is decoded and reads as unavailable. The second holds the review fixes: npm output is decoded with errors="replace"; any exception inside the gate is reported as unavailable (exit 2) with its traceback instead of exiting 1; every severity count must be a non-negative int, or the result is unavailable.

pipeline-scripts/npm_audit_gate.py (new)

  • It makes one npm audit --json --loglevel=http call.

  • When no usable report comes back, it retries up to NPM_AUDIT_MAX_ATTEMPTS times (default 3), NPM_AUDIT_RETRY_DELAY_SECONDS apart (default 15), with a per-attempt NPM_AUDIT_TIMEOUT_SECONDS (default 180). All three constants are env-backed, and an unset or invalid value falls back to the default.

  • It writes audit-results.json for the existing artifact upload.

  • It ends in one of three results, each with its own exit code, job-summary headline and ::error annotation:

    Result Exit Headline
    passed 0 "Passed: no high or critical advisories"
    found 1 "Failed, advisories found: N critical and M high" plus the counts table
    unavailable 2 "Failed, could not check: no usable audit report after k of N attempts: . This is not an advisory finding"
  • unavailable covers: no JSON, JSON that isn't a report, an {"error": …} object (its summary is quoted), no vulnerability counts, a timeout, and npm failing to start.

  • It stays fail-closed: a gate that didn't look never passes.

  • A found result is never retried.

  • Regression guard for the fallback. A report npm fetched through audits/quick counts as unavailable, with its own reason, so a future npm downgrade can't quietly depend on the retiring endpoint again.

  • It echoes npm's advisory-endpoint http lines into the job log, and the summary names the endpoint that answered.

frontend-test.yml, the Security Scan's "Run npm audit" step

Path filters. pipeline-scripts/npm_audit_gate.py is added to .github/filters/frontend-paths.yml and to its on.push.paths mirror, so a change to the gate runs the gate.

pipeline-scripts/npm_audit_gate_test.py (new) uses npm's own output shapes as fixtures: a v2 report, the verbatim 2026-09-11 error object, and bulk/quick http log lines. It covers:

  • the three results;
  • the unusable-report cases;
  • a report that came through audits/quick;
  • retry until success, stopping at the limit, and no retry on found;
  • a timeout;
  • the npm command and flags reaching subprocess.run;
  • the env constants and their fallbacks;
  • main()'s exit code, report file, summary headline and annotation for each result.

Verification

  • CI: pending on this PR. Because this PR touches frontend-test.yml, the Security Scan job runs on it. Its log is the evidence for the endpoint AC.
  • Local checks: only the commit's pre-commit hooks ran (black reformatted the two new files). No code from the repo was executed.

Acceptance criteria:

  • An audit endpoint error is retried, and if it persists, the job fails with a message naming the endpoint error, not advisories.
    • audit_with_retries retries only unavailable.
    • The reason quotes the error's summary, and the headline says "could not check… This is not an advisory finding".
    • Pinned by test_an_endpoint_error_is_retried_until_a_report_arrives, test_a_persistent_endpoint_error_stops_at_the_attempt_limit and test_main_states_which_result_happened[unavailable].
  • A high or critical finding still fails, labelled "advisories found" (AC2 as amended on ci(security): the npm audit gate fails an audit-endpoint error exactly like a high advisory — retry, and report 'could not check' distinctly #16337: the old wording was npm's output from the removed second call).
    • Exit 1, with "Failed, advisories found" and the counts. Pinned by test_a_high_or_critical_advisory_is_found and [found].
    • Honest note: the wording is now this gate's own headline, not npm's --audit-level text. The full report is still in the artifact.
  • The job summary states which of the three results happened. Every summary has a headline plus Result: <passed|found|unavailable>. Pinned by test_the_summary_says_could_not_check_distinctly_from_found.
  • A fixture test covers all three. npm_audit_gate_test.py.
  • The gate no longer depends on the retiring audits/quick endpoint.
    • The code-path finding and the npm 11 pin are above, and a quick-served report fails as unavailable.
    • The run log showing advisories/bulk in use is still to be cited from this PR's Security Scan job.

Related, not in scope: #16131 (the gate audits 1 of 13 package.json files) and #16260 (the terminal plugin path is still not mirrored in this file's push list).

Model Used

Claude Opus 5 (claude-opus-5)

…ort 'could not check' apart from advisories (#16337)

The Security Scan step called npm's audit service twice; the gate was the
second call, so an endpoint failure read exactly like a high advisory.
pipeline-scripts/npm_audit_gate.py makes one call (npm 11, which has no
audits/quick fallback), retries it while the service is unavailable, and
ends in passed / advisories found / could not check, each named in the job
summary and annotation. Fixture tests cover all three.
@coderabbitai

coderabbitai Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 403abbe1-8ae2-44b8-9fe9-a3f52510b3d0


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@mrveiss

mrveiss commented Sep 11, 2026

Copy link
Copy Markdown
Owner Author

Review at ed88877ff: the code passes. Two body-level actions remain. One commit and 4 files; the branch is 0 behind a12b4bf79d, includes #16300's base, and has no trailers.

Verified

  • One audit call. frontend-test.yml's Security Scan step now runs pipeline-scripts/npm_audit_gate.py once, with npm pinned through the step's environment (NPM_AUDIT_NPM_VERSION: '11.19.1'; 3 attempts, 15 s apart). The ci: the Security Scan check cannot fail on a security finding — every audit step ends in || true #13400 "this step gates" comment is kept and reworded. The gate script is added to .github/filters/frontend-paths.yml and to the workflow's own on.push.paths.
  • Three distinct results.
    • EXIT_CODES is passed 0, found 1, unavailable 2, and each failing result has its own annotation title (ANNOTATION_TITLES).
    • endpoint_used() reads npm's --loglevel=http lines, and audits/quick wins when both endpoints appear. classify() then treats a report that came through the quick endpoint as unavailable, never as a pass.
    • A timed-out attempt is unavailable.
    • Only an unavailable result is retried; a found result is not.
  • The tests cover every path (npm_audit_gate_test.py):
    • low and moderate advisories pass;
    • high and critical are found (parametrized);
    • an endpoint error is unavailable, and the message names the error;
    • a missing or unusable report is unavailable, never a pass;
    • a report that came through audits/quick is unavailable;
    • it retries until a report arrives, and stops at the attempt limit;
    • a found result is not retried, and a timeout is unavailable;
    • the environment configuration, including the fallback for an invalid value, works;
    • main states which result happened, and the summary tells "could not check" apart from "found".

Against #16337's criteria: AC1 (retry, then an endpoint-named failure), AC3 (the summary states which result) and AC4 (fixture tests for all three) are met in the code. The endpoint claim, that npm 11 uses advisories/bulk with no audits/quick fallback, is proven only once this PR's Security Scan log shows the advisories/bulk line. I haven't verified it from npm's source.

Action 1: AC2 says "unchanged", and the message changes. AC2 reads "A high or critical finding still fails with the advisory message, unchanged". The failing message is now the gate's own headline, not npm's --audit-level output. The reason holds: that text came from the second npm call, which is exactly what this PR removes. But the criterion should be amended on #16337 explicitly, with that reason, rather than satisfied in spirit.

Action 2: add a single-issue rationale. scripts/check_pr_issue_batching.py counts only issue references that follow a keyword (_REFERENCE, :43, applied at :86 after code spans are stripped). This body's only keyword reference is Closes #16337, and the other issue numbers are prose. So Check same-scope batching, still queued, will very likely fail, as it did on #16349. Add a line such as Single-issue rationale: …. Editing the body re-runs the check at the same head.

My four-layer pre-flight watcher is on ed88877ff.

@mrveiss

mrveiss commented Sep 11, 2026

Copy link
Copy Markdown
Owner Author

Second-session review of ed88877ff. Verdict: needs fixes. Two paths in the gate aren't fail-closed. Everything else checks out.

Severity Where Finding Scenario Fix
Medium-high pipeline-scripts/npm_audit_gate.py:169-175 (_attempt), :266-273 (main) _attempt catches only TimeoutExpired and OSError, and nothing wraps main(). subprocess.run(..., text=True) decodes with the locale codec and no errors=, so bytes that aren't UTF-8 raise an uncaught UnicodeDecodeError. npm writes a stray non-UTF-8 byte (a registry error page, say). The result is a traceback with no summary or annotation, and CPython's default exit 1, the same code as "advisories found". So a crash reads as a finding, which is exactly the ambiguity #16337 removes. Pass errors="replace", and/or catch everything else around the run as unavailable: keep the exception text, still write the summary and annotation, exit 2.
Medium npm_audit_gate.py:130-136 (_counts), :150-164 (classify) _counts turns a present but non-int severity value into 0, and a metadata.vulnerabilities of {} passes the isinstance(dict) check, so classify returns PASSED. A partial or malformed report, or a future schema emitting null, reads as "0 advisories", a pass. The gate fails open on data it never really got. Require every SEVERITY_ORDER key to be present with an int value; anything else is unavailable ("report carries incomplete vulnerability counts"), with a test.
Low npm_audit_gate_test.py:161,191 Only NPM_AUDIT_MAX_ATTEMPTS has invalid-value fallback tests. NPM_AUDIT_RETRY_DELAY_SECONDS is set only to a valid "0", and NPM_AUDIT_TIMEOUT_SECONDS isn't tested at all. A regression in _env_int's minimum= handling for delay or timeout goes unseen. Parametrize the fallback test over all three getters.

Verified:

  • The three outcomes stay distinct in exit code, headline and ::error title= annotation.
  • The retry is bounded (audit_with_retries :187-204, minimum=1, invalid values fall back to the default). It fires only on unavailable, never on found or passed, and a test pins that.
  • endpoint_used (:96-106) correctly prefers quick when both endpoints appear.
  • npm@11.19.1 is pinned exactly, and setup-node's cache: npm is kept.
  • permissions: contents: read is untouched.
  • The path-filter addition only widens the trigger to this one file.
  • Every function is under 30 lines.
  • _emit's print(...) # noqa: print matches the exemption ci_dispatch_watchdog.py already uses, so the baseline doesn't change.
  • pipeline-scripts is collected, via pytest.ini testpaths and ci.yml.

Unverified: the endpoint AC's run-log evidence (advisories/bulk). At review time all 29 checks were pending, and Security Scan hadn't reported yet. Check it once CI finishes.

Reviewed read-only, against PR head ed88877ff.

…check' (#16337)

Review of #16357: an undecodable byte in npm's output raised UnicodeDecodeError
and any uncaught exception exited 1, the 'advisories found' code, with no
summary; a missing or non-numeric severity count read as 0 and passed.
npm output is decoded with errors='replace', an exception inside the gate is
reported as unavailable (exit 2) with its traceback, and every severity
count must be a non-negative int. Fallback tests cover all three env constants.
@mrveiss

mrveiss commented Sep 11, 2026

Copy link
Copy Markdown
Owner Author

Delta review, ed88877ff→9174a08f0: passes. The commit is a fail-closed hardening of pipeline-scripts/npm_audit_gate.py. The branch is 0 behind and there are no trailers.

  • Malformed counts. _counts returns None when any severity count is missing, isn't an int, is a bool, or is negative, and classify then reports "could not check". Before, a malformed count read as 0 and could pass. The malformed-count path is tested: this commit adds report cases with missing, non-numeric, boolean or negative severity counts, which must read as "could not check".
  • Undecodable npm output. _run_npm_audit uses errors="replace", so undecodable bytes in npm's output can't raise. The report then fails to parse, which reads as "could not check".
  • The artifact write. _write_report can't decide the gate. A failed write of audit-results.json is logged and ignored.
  • The catch-all. _audit_or_could_not_check (:290-303) doesn't swallow errors. It logs the full traceback to stderr, puts the exception's type and message in the verdict, and returns UNAVAILABLE, exit 2, so the job still fails. The # noqa: BLE001 records why: an uncaught exception would exit 1, the code for "advisories found", so a gate bug would be mislabelled as an advisory. main routes through it (:309). The new test_a_crash_inside_the_gate_is_could_not_check_never_found covers it, and test_every_constant_comes_from_the_environment and test_an_invalid_constant_falls_back_to_its_default cover the configuration.

My four-layer pre-flight watcher has restarted on 9174a08f0.

@mrveiss

mrveiss commented Sep 11, 2026

Copy link
Copy Markdown
Owner Author

Delta review of ed88877ff..9174a08f0: all three earlier findings are fixed correctly. Two small test gaps remain, and the endpoint criterion's evidence is still unverified.

Confirmed by reading the code:

  • errors="replace" is set on subprocess.run (npm_audit_gate.py:172-182).
  • _audit_or_could_not_check (:290-303) wraps only audit_with_retries(...). It catches Exception, not BaseException, so SystemExit and KeyboardInterrupt still propagate, and a real FOUND (1) or PASSED (0) result is never remapped to 2.
  • A report or summary write that fails only warns. main takes the exit code from the outcome computed before the write, so a disk-full error can't change 0, 1 or 2.
  • _counts requires all five of npm's severity keys (critical, high, moderate, low, info) and ignores total. A missing key, a non-int, a bool or a negative number means "could not check".
  • Every new function is well under 30 lines, and all output still goes through _emit.
Severity Where Gap Fix
Low npm_audit_gate_test.py No test shows the exit code is unchanged when the report or summary write raises OSError. Monkeypatch the write to raise OSError, and assert the exit code still equals the verdict's.
Low npm_audit_gate_test.py:280 The crash test checks only capsys.readouterr().out, so the claim "traceback in the log" (stderr) is never asserted. Assert that the exception name or a traceback frame appears in .err.
Optional the tests errors="replace" together with audits/quick detection is sound, because the marker is ASCII, but no test mixes a bad byte with the endpoint line. Add a fixture with a non-UTF-8 byte next to a valid audits/quick log line.

Unverified: the endpoint criterion's advisories/bulk log evidence. The Frontend Testing Suite run (34616796835) at 9174a08f0 was still queued, and Security Scan hadn't started, so there's no log to grep. It gets checked on a later sweep, not treated as passing.

@github-actions

Copy link
Copy Markdown
Contributor

✅ SSOT Configuration Compliance: Passing

🎉 No new hardcoded values of either class — ssot and other both block.

Known backlog in pipeline-scripts/hardcoded_values_baseline.txt is suppressed and tracked in #14371.

…ode, and that crashes reach stderr (#16337)

Review of 9174a08: real OSErrors (report into a missing directory, summary
path a directory) leave passed/found/unavailable at 0/1/2; the crash test asserts
the traceback reaches stderr; a real subprocess emitting a non-UTF-8 byte beside
an audits/quick line is decoded (errors='replace') and reads as unavailable.
@mrveiss

mrveiss commented Sep 11, 2026

Copy link
Copy Markdown
Owner Author

Delta 9174a08f0..d5f1bfae2 (tests only): approve. The two gaps and the optional case from my last review are closed.

  • test_a_failed_report_or_summary_write_never_changes_the_exit_code is parametrized over passed, found and unavailable. It raises real OSErrors (no mocks) and asserts gate.main(...) == gate.EXIT_CODES[verdict], plus both "Could not write" lines on stderr.
  • The crash test now asserts that Traceback and the exception's type name appear in .err.
  • test_undecodable_npm_output_is_decoded_not_raised runs a real subprocess through _run_npm_audit. A \\xff byte next to an audits/quick line comes back as U+FFFD, and the verdict is (UNAVAILABLE, "audits/quick").

Still open, before the train: the endpoint criterion's advisories/bulk evidence from the Security Scan log.

@github-actions

Copy link
Copy Markdown
Contributor

Notice: 29 open PRs — past the runaway threshold (25)

There is no PR queue limit, and this is not a request to defer this PR. Work proceeds one issue at a time without a cap on open PRs; review capacity is the constraint.

This notice only means the count is high enough to be worth a glance for a runaway — something opening PRs in a loop, or a merge pipeline that has stalled so nothing is draining.

Currently open:

If the queue is draining normally, ignore this. Otherwise:

  1. Check whether CI is dispatching at all — see the ci-dispatch-watchdog status on these PRs
  2. Merge the ones whose CI has finished and review has passed: gh pr merge <number> --squash --delete-branch
  3. Look for a loop opening near-identical PRs

Warn-only runaway detector — .github/workflows/pr-queue-gate.yml. It never blocks a merge.

@mrveiss

mrveiss commented Sep 11, 2026

Copy link
Copy Markdown
Owner Author

Delta review, 9174a08f0→d5f1bfae2: passes. The commit is test-only (npm_audit_gate_test.py, +51/−1), with no trailers, and the branch is 0 behind. It pins what the previous review relied on:

  • A crash reaches the job log. test_a_crash_inside_the_gate_is_could_not_check_never_found now also asserts that Traceback and the exception type appear on stderr, so the crash is labelled, not swallowed.
  • A failed write never moves the exit code. test_a_failed_report_or_summary_write_never_changes_the_exit_code uses real OSErrors: the report goes into a directory that doesn't exist, and GITHUB_STEP_SUMMARY points at a directory. It checks that the exit code still equals the verdict's for passed, found and unavailable, and that both "Could not write …" lines reach stderr.
  • Undecodable output doesn't raise. test_undecodable_npm_output_is_decoded_not_raised runs a real subprocess that writes a non-UTF-8 byte beside an audits/quick log line. errors=\"replace\" hands classify() text, and the quick endpoint still reads as "could not check".

My pre-flight watcher has restarted on d5f1bfae2.

mrveiss added a commit that referenced this pull request Sep 12, 2026
mrveiss added a commit that referenced this pull request Sep 12, 2026
@mrveiss mrveiss added this to the v0.9.0 milestone Sep 12, 2026
@mrveiss
mrveiss merged commit ad81083 into Dev_new_gui Sep 12, 2026
67 of 71 checks passed
@mrveiss
mrveiss deleted the issue-16337 branch September 12, 2026 09:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant