Skip to content

feat(ci): run secreview headless on PRs and post findings as a comment #12973

Description

@mrveiss

Problem

Security review of a diff is currently a manual chore that has to be initiated,
watched, and waited on. Usage analysis over 59 sessions found it was the single
most-requested task (11 sessions) and that 8 of the 11 delivered nothing —
abandoned mid-exploration. #12951 fixed the ordering with the secreview
skill, but the review still only happens if a human remembers to ask for one.

The skill proved its value immediately: dogfooded against PR #12950 it found two
confirmed defects (#12956) — and that PR merged anyway, because the review was
an out-of-band comment rather than a gate.

What existing CI does and does not cover

.github/workflows/security.yml is mature on static tooling: pip-audit, safety,
npm audit, bandit, semgrep, secret detection, SARIF upload to the Security tab.
None of that is duplicated here.

What pattern matchers structurally cannot catch is what the manual reviews kept
finding — semantic defects that need the diff read in context:

There is currently no reasoning-based review anywhere in CI:

$ grep -rln "claude" .github/workflows/
(no matches)

Proposed fix

A pull_request workflow that runs the secreview skill headless and posts its
findings table as a PR comment.

Cost control — reviews are billed per run, and credit exhaustion is already a
known failure mode, so the job must be narrow by default:

  • path-filtered to backend and auth-adjacent code, not every PR
  • skipped on draft PRs
  • output capped; the skill's contract is already a table plus a verdict
  • advisory (comment only), not a required check — findings inform the human gate
    rather than blocking merge on a probabilistic reviewer

Must be inert-safe. The repo currently has no secrets configured at all
(gh api repos/.../actions/secrets → total_count: 0), so ANTHROPIC_API_KEY
does not exist yet. The workflow must skip cleanly when the secret is absent,
so merging it cannot red every PR before the key is added.

Prerequisite (owner action)

ANTHROPIC_API_KEY must be added as a repository secret before the job does
anything. Until then it is expected to skip.

Acceptance criteria

  • Workflow runs on ubuntu-latest, not the self-hosted singleton runner
  • Skips cleanly with a visible notice when ANTHROPIC_API_KEY is unset
  • Skips draft PRs and PRs touching no filtered paths
  • Posts exactly one comment per PR, updated in place rather than appended per push
  • Advisory only — does not become a required check
  • Passes actionlint
  • Does not duplicate any scan already in security.yml

Activity

  1. github-actions commented on Jul 29, 2026

    @github-actions
    Contributor

    PR #12974 (merged to Dev_new_gui) references this issue with a close keyword.

    feat(ci): run secreview headless on PRs and post findings as a comment (#12973)

    If this issue is fully resolved, close it manually. If work remains, no action is needed.

  2. mrveiss commented on Jul 29, 2026

    @mrveiss
    OwnerAuthor

    Delivered in PR #12974, squash-merged to Dev_new_gui as eaf1b3923.

    $ gh pr view 12974 --json state,mergedAt --jq '.state, .mergedAt'
    MERGED
    2026-07-29T19:35:44Z
    
    $ git ls-tree -r origin/Dev_new_gui --name-only | grep ai-security-review
    .github/workflows/ai-security-review.yml
    

    CI: 27 checks passed, 0 failed.

    The carried pre-existing actionlint fix is also on base:

    $ git show origin/Dev_new_gui:.github/workflows/api-wiring.yml | sed -n '23,27p'
        # No YAML anchor here: #12934 removed the pull_request.paths filter that
        # used to alias it, so naming the anchor now fails actionlint's
        # syntax-check as "defined but not used".
        paths:
          - 'autobot-backend/**/*.py'
    

    Still inert — owner action outstanding

    The workflow is merged but does nothing yet. ANTHROPIC_API_KEY does not exist
    as a repository secret, so the job skips with a ::notice:: by design. Until it
    is added, the model-invocation path remains untested in CI — the acceptance
    criteria covering skip-safety, path filtering, and actionlint are met, but
    "posts a findings comment" has never actually executed.

    Closing on delivery of the workflow itself. Enabling it is a one-step owner
    action, not remaining engineering work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions