Repository navigation
fix(ci): gate the frontend npm audit on one report, retry it, and report 'could not check' apart from advisories (#16337) - #16357
Conversation
…ort 'could not check' apart from advisories (#16337) The Security Scan step called npm's audit service twice; the gate was the second call, so an endpoint failure read exactly like a high advisory. pipeline-scripts/npm_audit_gate.py makes one call (npm 11, which has no audits/quick fallback), retries it while the service is unavailable, and ends in passed / advisories found / could not check, each named in the job summary and annotation. Fixture tests cover all three.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Review at Verified
Against #16337's criteria: AC1 (retry, then an endpoint-named failure), AC3 (the summary states which result) and AC4 (fixture tests for all three) are met in the code. The endpoint claim, that npm 11 uses Action 1: AC2 says "unchanged", and the message changes. AC2 reads "A high or critical finding still fails with the advisory message, unchanged". The failing message is now the gate's own headline, not npm's Action 2: add a single-issue rationale. My four-layer pre-flight watcher is on |
|
Second-session review of
Verified:
Unverified: the endpoint AC's run-log evidence ( Reviewed read-only, against PR head |
…check' (#16337) Review of #16357: an undecodable byte in npm's output raised UnicodeDecodeError and any uncaught exception exited 1, the 'advisories found' code, with no summary; a missing or non-numeric severity count read as 0 and passed. npm output is decoded with errors='replace', an exception inside the gate is reported as unavailable (exit 2) with its traceback, and every severity count must be a non-negative int. Fallback tests cover all three env constants.
|
Delta review,
My four-layer pre-flight watcher has restarted on |
|
Delta review of Confirmed by reading the code:
Unverified: the endpoint criterion's |
✅ SSOT Configuration Compliance: Passing🎉 No new hardcoded values of either class — Known backlog in |
…ode, and that crashes reach stderr (#16337) Review of 9174a08: real OSErrors (report into a missing directory, summary path a directory) leave passed/found/unavailable at 0/1/2; the crash test asserts the traceback reaches stderr; a real subprocess emitting a non-UTF-8 byte beside an audits/quick line is decoded (errors='replace') and reads as unavailable.
|
Delta
Still open, before the train: the endpoint criterion's |
|
Delta review,
My pre-flight watcher has restarted on |
Closes #16337
Single-issue rationale: this changes only the Security Scan's audit gate (one workflow step, its script and its path filter). The related issues change what the gate covers or what triggers it, which is a different risk: #16131 (audit the other 12 package.json files) and #16260 (the terminal plugin push path). They ride separately.
Thinking Path
npm audit --json > audit-results.json || truefetched the report. That call succeeded.npm audit --audit-level=highre-fetched the same data as the gate, and that second call was the one that got the 400.workspaces/arborist/lib/audit-report.js:323-336tries/-/npm/v1/security/advisories/bulkand, when that fails, falls back to/-/npm/v1/security/audits/quick. That quick endpoint answered "this endpoint is being retired".audits/quickreference at all. npm 11 supports Node^20.17.0 || >=22.9.0.What Changed
Commits ed88877, 9174a08 and d5f1bfa. The third adds the review's test gaps: real OSErrors on the report and summary writes leave the exit code equal to the verdict's; the crash test asserts the traceback reaches stderr; a real subprocess emitting a non-UTF-8 byte next to an
audits/quickline is decoded and reads as unavailable. The second holds the review fixes: npm output is decoded witherrors="replace"; any exception inside the gate is reported asunavailable(exit 2) with its traceback instead of exiting 1; every severity count must be a non-negative int, or the result isunavailable.pipeline-scripts/npm_audit_gate.py(new)It makes one
npm audit --json --loglevel=httpcall.When no usable report comes back, it retries up to
NPM_AUDIT_MAX_ATTEMPTStimes (default 3),NPM_AUDIT_RETRY_DELAY_SECONDSapart (default 15), with a per-attemptNPM_AUDIT_TIMEOUT_SECONDS(default 180). All three constants are env-backed, and an unset or invalid value falls back to the default.It writes
audit-results.jsonfor the existing artifact upload.It ends in one of three results, each with its own exit code, job-summary headline and
::errorannotation:passedfoundunavailableunavailablecovers: no JSON, JSON that isn't a report, an{"error": …}object (itssummaryis quoted), no vulnerability counts, a timeout, and npm failing to start.It stays fail-closed: a gate that didn't look never passes.
A
foundresult is never retried.Regression guard for the fallback. A report npm fetched through
audits/quickcounts asunavailable, with its own reason, so a future npm downgrade can't quietly depend on the retiring endpoint again.It echoes npm's advisory-endpoint http lines into the job log, and the summary names the endpoint that answered.
frontend-test.yml, the Security Scan's "Run npm audit" stepnpx --yes npm@${NPM_AUDIT_NPM_VERSION}, withNPM_AUDIT_NPM_VERSION: '11.19.1'pinned in the step's env.Path filters.
pipeline-scripts/npm_audit_gate.pyis added to.github/filters/frontend-paths.ymland to itson.push.pathsmirror, so a change to the gate runs the gate.pipeline-scripts/npm_audit_gate_test.py(new) uses npm's own output shapes as fixtures: a v2 report, the verbatim 2026-09-11 error object, and bulk/quick http log lines. It covers:audits/quick;found;subprocess.run;main()'s exit code, report file, summary headline and annotation for each result.Verification
frontend-test.yml, the Security Scan job runs on it. Its log is the evidence for the endpoint AC.Acceptance criteria:
audit_with_retriesretries onlyunavailable.summary, and the headline says "could not check… This is not an advisory finding".test_an_endpoint_error_is_retried_until_a_report_arrives,test_a_persistent_endpoint_error_stops_at_the_attempt_limitandtest_main_states_which_result_happened[unavailable].test_a_high_or_critical_advisory_is_foundand[found].--audit-leveltext. The full report is still in the artifact.Result: <passed|found|unavailable>. Pinned bytest_the_summary_says_could_not_check_distinctly_from_found.npm_audit_gate_test.py.audits/quickendpoint.unavailable.advisories/bulkin use is still to be cited from this PR's Security Scan job.Related, not in scope: #16131 (the gate audits 1 of 13
package.jsonfiles) and #16260 (the terminal plugin path is still not mirrored in this file's push list).Model Used
Claude Opus 5 (
claude-opus-5)