Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion .claude/pr-review-agent/ADVISORY_REVIEW_GATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -217,18 +217,28 @@ Excluding: CodeRabbit rate-limits, PRs blocked by CodeRabbit

### Bot Customization

To add/remove bots, edit the `ADVISORY_BOTS` map:
To add/remove bots, edit the `ADVISORY_BOTS` map (code in
`scripts/lib/advisory-review-gate.sh` remains the source of truth; this block is
illustrative and mirrors the current registry):

```bash
declare -A ADVISORY_BOTS=(
[gemini-code-assist]="Gemini Code Assist (advisory)"
[copilot-pull-request-reviewer]="Copilot PR Reviewer (advisory)"
[sonarqubecloud]="SonarCloud (advisory)"
[chatgpt-codex-connector]="Codex (advisory, newer bot)"
[qodo-code-review]="Qodo Merge (advisory)"
[codeant-ai]="CodeAnt (advisory)"
[graphite-app]="Graphite (advisory)"
# Add new bots here
)
Comment thread
coderabbitai[bot] marked this conversation as resolved.
```

CodeRabbit (`coderabbitai`) is deliberately **not** in `ADVISORY_BOTS` — the gate
does not wait on it (see [CodeRabbit Status](#coderabbit-status)) — but it is
tracked by the reviewer scorecard, so it appears in `RATE_LIMIT_NOTICE_BOTS` /
`scripts/reviewer_report.sh` alongside the bots above.

## Testing

Run the advisory gate tests:
Expand Down
3 changes: 2 additions & 1 deletion docs/reviewer-report.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Org-wide, **deterministic** visibility into the agentic third-party code reviewers that
participate in PR review across `petry-projects` — GitHub Copilot, Gemini Code Assist,
Codex, CodeRabbit, SonarCloud, Qodo Merge, and CodeAnt. Every figure is computed with `jq`/`awk` from GitHub's
Codex, CodeRabbit, SonarCloud, Qodo Merge, CodeAnt, and Graphite. Every figure is computed with `jq`/`awk` from GitHub's
own review data. **No LLM is involved** in this pipeline: a narrative / quality-scoring
layer is a deliberately separate, human-approved add-on (see [Roadmap](#roadmap-v2--llm-optional-human-approved)).

Expand Down Expand Up @@ -30,6 +30,7 @@ drift from the approval gate's notion of who these bots are.
| SonarCloud | `sonarqubecloud` |
| Qodo Merge | `qodo-code-review` |
| CodeAnt | `codeant-ai` |
| Graphite | `graphite-app` |

The org's own Claude reviewer (`donpetry-bot`) is intentionally **out of scope** here — its
cost lives in the [Token Cost Observatory](./token-report.md); this report is about the
Expand Down
2 changes: 2 additions & 0 deletions scripts/lib/advisory-review-gate.sh
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ declare -Ar ADVISORY_BOTS=(
[chatgpt-codex-connector]="Codex (advisory, newer bot)"
[qodo-code-review]="Qodo Merge (advisory)"
[codeant-ai]="CodeAnt (advisory)"
[graphite-app]="Graphite (advisory)"
Comment thread
don-petry marked this conversation as resolved.
)

# Canonical rate-limit / out-of-quota body pattern — the SINGLE source of truth
Expand Down Expand Up @@ -192,6 +193,7 @@ declare -ar RATE_LIMIT_NOTICE_BOTS=(
coderabbitai
qodo-code-review
codeant-ai
graphite-app
)

# Case-insensitive phrases that indicate a bot is itself rate-limited / out of
Expand Down
7 changes: 4 additions & 3 deletions scripts/reviewer_report.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
# reviewer_report.sh — org-wide Third-Party Reviewer Scorecard.
#
# Measures the agentic third-party code reviewers (GitHub Apps) that participate
# in PR review across the org — Copilot, Gemini Code Assist, Codex, CodeRabbit,
# and SonarCloud — and renders a DETERMINISTIC weekly Markdown report. No LLM is
# in PR review across the org and renders a DETERMINISTIC weekly Markdown report.
# No LLM is
# used anywhere in this pipeline: every metric is computed with jq/awk/bash from
# GitHub's own review data. (An LLM narrative is a deliberately separate, human-
# triggered add-on — not part of this script.)
Expand Down Expand Up @@ -69,7 +69,7 @@ fi
if declare -p RATE_LIMIT_NOTICE_BOTS >/dev/null 2>&1 && [ "${#RATE_LIMIT_NOTICE_BOTS[@]}" -gt 0 ]; then
REVIEWER_BOTS=("${RATE_LIMIT_NOTICE_BOTS[@]}")
else
REVIEWER_BOTS=(gemini-code-assist copilot-pull-request-reviewer sonarqubecloud chatgpt-codex-connector coderabbitai qodo-code-review codeant-ai)
REVIEWER_BOTS=(gemini-code-assist copilot-pull-request-reviewer sonarqubecloud chatgpt-codex-connector coderabbitai qodo-code-review codeant-ai graphite-app)
Comment thread
qodo-code-review[bot] marked this conversation as resolved.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
fi

# Human-facing display names, keyed by GraphQL login (no "[bot]" suffix).
Expand All @@ -85,6 +85,7 @@ declare -gA REVIEWER_LABELS=(
[sonarqubecloud]="SonarCloud"
[qodo-code-review]="Qodo Merge"
[codeant-ai]="CodeAnt"
[graphite-app]="Graphite"
)

# Rate-limit / out-of-quota body pattern — reuse the gate's if present.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"reviews":[],"comments":[{"author":{"login":"graphite-app"},"createdAt":"2099-01-01T00:00:00Z","body":"Graphite has reached your monthly usage limit for pull-request reviews on this repository. Reviews will resume when the limit resets."}]}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"reviews":[{"author":{"login":"graphite-app"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z","body":"Graphite review: 2 suggestions found on this change."}],"comments":[]}
62 changes: 54 additions & 8 deletions tests/dev-lead/unit/test_advisory_review_gate.bats
Original file line number Diff line number Diff line change
Expand Up @@ -525,7 +525,7 @@ MOCK_EOF
# Three real advisory reviews + Codex signalling it is out of quota, head + submissions
# recent (timeout fallbacks disarmed). All available bots have submitted, so the gate approves.
local json
json='{"reviews":[{"author":{"login":"gemini-code-assist"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"},{"author":{"login":"sonarqubecloud"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"},{"author":{"login":"copilot-pull-request-reviewer"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"},{"author":{"login":"qodo-code-review"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"},{"author":{"login":"codeant-ai"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"}],"comments":[{"author":{"login":"chatgpt-codex-connector"},"body":"You have reached your Codex usage limits for code reviews.","createdAt":"2099-01-01T00:00:00Z"}]}'
json='{"reviews":[{"author":{"login":"gemini-code-assist"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"},{"author":{"login":"sonarqubecloud"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"},{"author":{"login":"copilot-pull-request-reviewer"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"},{"author":{"login":"qodo-code-review"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"},{"author":{"login":"codeant-ai"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"},{"author":{"login":"graphite-app"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"}],"comments":[{"author":{"login":"chatgpt-codex-connector"},"body":"You have reached your Codex usage limits for code reviews.","createdAt":"2099-01-01T00:00:00Z"}]}'
local tmpdir
tmpdir=$(_make_mock_gh_dir_recent "$json")
local gate_script="$SCRIPT_DIR/lib/advisory-review-gate.sh"
Expand Down Expand Up @@ -699,12 +699,12 @@ MOCK_EOF
# ────────────────────────────────────────────────────────────────────
# NEW ADVISORY REVIEWERS — Qodo Merge + CodeAnt (issue #1349)
#
# Graphite is intentionally NOT registered: it had authored no review/comment
# at implementation time, so its GraphQL author login is unverified (the app
# slug `graphite-app` is not the same identifier namespace as `.author.login`,
# and the Qodo guess `qodo-merge-pro` was already proven wrong — real login is
# `qodo-code-review`). Adding a guessed login would violate the "do not guess
# identifiers" guardrail; it is held pending its first real review.
# Graphite was deferred here because it had authored no review/comment at
# #1349/#1355 implementation time, so its GraphQL author login was unverified
# (the Qodo guess `qodo-merge-pro` was already proven wrong — real login is
# `qodo-code-review`). It has since posted real COMMENTED reviews on PR #1355,
# verifying its login is `graphite-app`, and is now registered under #1401
# (see the Graphite section below).
# ────────────────────────────────────────────────────────────────────

_events_dir() {
Expand Down Expand Up @@ -815,7 +815,7 @@ _events_dir() {
# Qodo is out of quota; all other advisory bots have submitted real reviews.
# The rate-limited Qodo must drop out of the required set so the gate approves.
local json
json='{"reviews":[{"author":{"login":"gemini-code-assist"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"},{"author":{"login":"copilot-pull-request-reviewer"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"},{"author":{"login":"sonarqubecloud"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"},{"author":{"login":"chatgpt-codex-connector"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"},{"author":{"login":"codeant-ai"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"}],"comments":[{"author":{"login":"qodo-code-review"},"body":"Qodo Merge has reached your monthly usage limit for pull-request reviews.","createdAt":"2099-01-01T00:00:00Z"}]}'
json='{"reviews":[{"author":{"login":"gemini-code-assist"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"},{"author":{"login":"copilot-pull-request-reviewer"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"},{"author":{"login":"sonarqubecloud"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"},{"author":{"login":"chatgpt-codex-connector"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"},{"author":{"login":"codeant-ai"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"},{"author":{"login":"graphite-app"},"state":"COMMENTED","submittedAt":"2099-01-01T00:00:00Z"}],"comments":[{"author":{"login":"qodo-code-review"},"body":"Qodo Merge has reached your monthly usage limit for pull-request reviews.","createdAt":"2099-01-01T00:00:00Z"}]}'
local tmpdir; tmpdir=$(_make_mock_gh_dir_recent "$json")
local gate_script="$SCRIPT_DIR/lib/advisory-review-gate.sh"
run env PATH="$tmpdir:$PATH" bash -c "
Expand All @@ -825,3 +825,49 @@ _events_dir() {
rm -rf "$tmpdir"
[ "$status" -eq 0 ]
}

# ────────────────────────────────────────────────────────────────────
# GRAPHITE ADVISORY REVIEWER (issue #1401, follow-up to #1349 / #1355)
#
# Deferred in #1355 because Graphite had authored no review/comment, so its
# GraphQL author login was unverified. It has since posted real COMMENTED
# reviews on PR #1355, confirming its login is `graphite-app` (the app slug
# happens to equal the .author.login here) — so it is now registered per the
# "do not guess identifiers" guardrail (login verified, not guessed).
# ────────────────────────────────────────────────────────────────────

@test "Advisory gate: Graphite is a registered advisory bot (issue #1401)" {
grep -q 'graphite-app' "$SCRIPT_DIR/lib/advisory-review-gate.sh"
}

@test "Advisory gate: Graphite is in the rate-limit notice superset (issue #1401)" {
run bash -c "source '$SCRIPT_DIR/lib/advisory-review-gate.sh'
printf '%s\n' \"\${RATE_LIMIT_NOTICE_BOTS[@]}\""
[[ "$output" == *"graphite-app"* ]]
}

@test "Gate runtime: a Graphite review is detected as an advisory bot submission (issue #1401)" {
local json; json="$(cat "$(_events_dir)/advisory_graphite_reviewed.json")"
local tmpdir; tmpdir=$(_make_mock_gh_dir_recent "$json")
local gate_script="$SCRIPT_DIR/lib/advisory-review-gate.sh"
run env PATH="$tmpdir:$PATH" bash -c "
source '$gate_script'
check_advisory_reviews 'https://github.com/owner/repo/pull/123'
"
rm -rf "$tmpdir"
# If Graphite were unregistered it would be filtered out and never appear in output.
[[ "$output" == *"graphite-app"* ]]
}

@test "Gate runtime: a Graphite out-of-quota notice is classified RATE_LIMITED (issue #1401)" {
local json; json="$(cat "$(_events_dir)/advisory_graphite_rate_limited.json")"
local tmpdir; tmpdir=$(_make_mock_gh_dir_recent "$json")
local gate_script="$SCRIPT_DIR/lib/advisory-review-gate.sh"
run env PATH="$tmpdir:$PATH" bash -c "
source '$gate_script'
check_advisory_reviews 'https://github.com/owner/repo/pull/123'
"
rm -rf "$tmpdir"
[[ "$output" == *"RATE_LIMITED"* ]]
[[ "$output" == *"graphite-app"* ]]
}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
10 changes: 8 additions & 2 deletions tests/reviewer_report.bats
Original file line number Diff line number Diff line change
Expand Up @@ -17,20 +17,26 @@ setup() {
# Registry wiring — bots come from the shared advisory-review-gate list
# ---------------------------------------------------------------------------

@test "REVIEWER_BOTS: seven tracked reviewers, sourced from the gate registry" {
[ "${#REVIEWER_BOTS[@]}" -eq 7 ]
@test "REVIEWER_BOTS: eight tracked reviewers, sourced from the gate registry" {
[ "${#REVIEWER_BOTS[@]}" -eq 8 ]
[[ " ${REVIEWER_BOTS[*]} " == *" coderabbitai "* ]]
[[ " ${REVIEWER_BOTS[*]} " == *" copilot-pull-request-reviewer "* ]]
# Qodo Merge + CodeAnt registered via the shared gate registry (issue #1349).
[[ " ${REVIEWER_BOTS[*]} " == *" qodo-code-review "* ]]
[[ " ${REVIEWER_BOTS[*]} " == *" codeant-ai "* ]]
# Graphite registered via advisory-review-gate (issue #1401).
[[ " ${REVIEWER_BOTS[*]} " == *" graphite-app "* ]]
}

@test "REVIEWER_LABELS: Qodo Merge + CodeAnt have display names (issue #1349)" {
[ "${REVIEWER_LABELS[qodo-code-review]}" = "Qodo Merge" ]
[ "${REVIEWER_LABELS[codeant-ai]}" = "CodeAnt" ]
}

@test "REVIEWER_LABELS: Graphite has display name (issue #1401)" {
[ "${REVIEWER_LABELS[graphite-app]}" = "Graphite" ]
}

@test "REVIEWER_LABELS: every tracked reviewer has a display name (no drift, issue #1349)" {
# The report list is derived from the gate list; each tracked login must carry a
# human-facing label so no reviewer renders as a bare GraphQL login.
Expand Down
Loading