Skip to content

Report incomplete PR descriptions before review - #2371

Merged
kriszyp merged 7 commits into
mainfrom
kris/pr-format-check
Aug 31, 2026
Merged

kriszyp merged 7 commits into
mainfrom
kris/pr-format-check

Conversation

@kriszyp

@kriszyp kriszyp commented Aug 28, 2026 •

Copy link
Copy Markdown
Member

Adds a report-only PR-description evaluator to the existing review-coverage workflow so substantive engineer-authored PRs expose missing structure and stale or absent current-diff links before review. The current workflow remains informational and read-only; enforcement is explicitly sequenced behind trusted execution.

For the human reviewer

  1. Trusted enforcement host: this PR runs from the PR checkout in report mode; a later pull_request_target or equivalent base-revision workflow must host the action before format_mode: enforce is credible. Reversing this choice means expanding this PR's trust boundary now; declining it leaves the check informational.
  2. Unavailable evidence: GitHub-omitted patches, incomplete file lists, and collection failures are warnings in report mode but noncompliant in a future enforce mode; the normalized API artifact is bounded and schema-validated. This is locally reversible; choosing neutral enforcement would let API gaps bypass link freshness.
  3. All links current: every recognized PR-diff link must target this PR and resolve to current hunk endpoints, rather than accepting one good link alongside stale context. This is a small policy change if the author friction proves too high.
  4. Syntax-only evidence: the section evaluator proves that Verification and reviewer sections have visible content and correct ordering; it does not decide whether the claimed test or rationale is true. Stronger semantics would need a structured evidence contract, not more Markdown heuristics.

Verification

  • node --test .github/actions/review-coverage/*.test.mjs — 83 passed, 0 failed.
  • npx prettier --write .github/actions/review-coverage/*.mjs .github/actions/review-coverage/action.yml .github/workflows/review-coverage.yml and npm run lint:required — passed.
  • npm run build and npm run test:unit:resources — passed; resource suite reported 1,767 passing and 22 pending.
  • Live smoke against Harper PR Apply audit retention continuously to RocksDB transaction logs #2338's body and Files API artifact passed; removing its links without changing the head failed as intended. The pinned resources/auditStore.ts anchor was also verified against that live PR body.
  • The existing main CLI accepted the same-head linkless regression fixture; this branch rejected it.
  • npm run test:integration:all completed with 1,834 passing, 18 skipped, 3 unrelated runtime failures, and 7 cancellations under the failing QA-720 startup arm. npm run test:unit:main was blocked by an existing deploy-by-reference fixture requiring a remote branch and a lingering process-group harness; neither gate touches .github code.

Complexity: complicated

Review-Coverage: authored=codex; ran=claude,gemini; declined=cursor-grok,cursor-composer,domain; rounds=7 @ 5e56d86

Human-Review-Need: 3 @ 5e56d86

@kriszyp
kriszyp requested review from cb1kenobi and heskew August 28, 2026 00:05
@kriszyp kriszyp added this to the v5.3 milestone Aug 28, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a PR description format checker to the review-coverage GitHub action, validating summary prose, verification sections, line-anchored links, and AI-related fields. Feedback on the changes highlights an issue in inspectBodyText where omitting lines inside code blocks shifts character offsets and causes false positives for verification sections containing only code. Additionally, suggestions were made to use loose equality (== null and != null) for null-or-undefined checks to comply with the repository's style guide.

Comment thread .github/actions/review-coverage/prFormatLinks.mjs
Comment thread .github/actions/review-coverage/prFormatLinks.mjs Outdated
Comment thread .github/actions/review-coverage/prFormatLinks.mjs Outdated
@kriszyp
kriszyp marked this pull request as ready for review August 28, 2026 11:59
Comment on lines +24 to +30
if: >-
github.event.pull_request.user.type != 'Bot' &&
!endsWith(github.event.pull_request.user.login, '[bot]') &&
(github.event.pull_request.author_association == 'MEMBER' || github.event.pull_request.author_association == 'OWNER') &&
(github.event.pull_request.additions > 2 || github.event.pull_request.deletions > 2 ||
(github.event.pull_request.additions == 2 && github.event.pull_request.deletions > 0) ||
(github.event.pull_request.deletions == 2 && github.event.pull_request.additions > 0))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion (non-blocking): This if: condition re-derives isTrivialChange's additions + deletions <= 2 threshold as a manual boolean decomposition (GHA expressions have no + operator), duplicating the invariant in reviewGate.mjs. Nothing ties the two together — if the threshold in isTrivialChange ever changes, this condition silently drifts out of sync and no test catches it (the workflow-shape test only checks for the author_association == 'MEMBER' substring, not the arithmetic). Consider a one-line comment here pointing at reviewGate.mjs#isTrivialChange so a future threshold change is caught, or add a test enumerating (additions, deletions) pairs asserting this expression matches isTrivialChange.

@claude

claude Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Reviewed; no blockers found. One non-blocking suggestion posted inline.

@kriszyp
kriszyp merged commit 7225fdf into main Aug 31, 2026
58 checks passed
@kriszyp
kriszyp deleted the kris/pr-format-check branch August 31, 2026 13:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant