pr-review never auto-approves when advisory bots are rate-limited/out-of-quota — advisory-gate blocks on bots that can't submit

[don-petry]

Summary

The pr-review agent is not failing — it is skipping auto-approval on green PRs with decision: skip, reason: waiting-for-advisory-bots, and the skip never clears. Result: clean, CI-green, rebased PRs sit at REVIEW_REQUIRED indefinitely because donpetry-bot never posts its org-leads approval.

Root cause: the advisory-review-gate waits for advisory bots to submit a review at head, but two of the four advisory bots are out of quota and cannot submit, and the gate does not degrade gracefully when a bot is rate-limited.

Evidence

pr-review run for PR #623 (review/review job 81223083427, run 27478909009):

[advisory-gate] Advisory bots detected: gemini-code-assist sonarqubecloud
[advisory-gate] WARNING: Only 2/4 advisory bots submitted so far (head age: ~60s, last submission: ~Ns ago)
[advisory-gate] WARNING: Will re-check when remaining bots submit their reviews
skip: advisory bots still reviewing (non-blocking, will re-check on bot submission)
{"pr":".../623","decision":"skip","reason":"waiting-for-advisory-bots"}

The two missing/non-responding advisory bots are out of quota — confirmed by their own comments across #479, #483, #516, #518, #520, #525, #550, #580, #621, #622, #623:

• CodeRabbit: "Review limit reached … Your organization has used up its prepaid credits, and credit purchases are no longer available."
• chatgpt-codex-connector: "You have reached your Codex usage limits for code reviews."

Root cause (code)

scripts/lib/advisory-review-gate.sh:

• ADVISORY_BOTS (L32–37) = gemini-code-assist, copilot-pull-request-reviewer, sonarqubecloud, chatgpt-codex-connector.
• check_advisory_reviews (L128–235) requires num_submitted == total_advisory_bots (all 4) or one of two timeout fallbacks: head-commit age > 1200s (L222) or quiescence > 600s (L224). Otherwise return 1 → pr-review skips (review-one-pr.sh L140–177, reason=waiting-for-advisory-bots).
• In practice the gate rarely sees 4/4: Codex is permanently out of quota, and Copilot reviews only a subset of PRs. So approval depends entirely on the timeout fallbacks — which only re-evaluate on an event re-trigger (a bot submission) or the hourly stuck-review sweep. Every dev-lead push (e.g. a rebase) resets the head-age clock to 0, so freshly-rebased PRs restart the 20-min wait and keep skipping.
• A RATE_LIMITED display state already exists (format_bot_status L110) but nothing ever sets it — get_advisory_bot_states (L59–85) only maps a bot's literal review/comment state. So the half-built rate-limit handling never triggers.

Impact

This is the gate behind the current backlog of green-CI PRs stuck at REVIEW_REQUIRED (e.g. #479, #483, #516, #518, #520, #525, #550, #589, #620, #621, #622, #623). With CodeRabbit + Codex credits exhausted, the advisory reviews won't arrive, so the gate only clears via the 20-min/10-min fallbacks after a quiet period + a re-trigger — slow and fragile, and reset by any new push.

Recommended fix (pick, keep minimal + tests)

1. Wire up RATE_LIMITED (preferred). In get_advisory_bot_states, detect a bot comment whose body matches the known rate-limit/usage-limit markers (CodeRabbit "Review limit reached"/"used up its prepaid credits"; Codex "reached your Codex usage limits") and emit state: RATE_LIMITED. Treat RATE_LIMITED (and UNSUPPORTED) bots as responded/non-participating so they don't hold the gate. This makes the gate degrade gracefully the moment a bot signals it's out.
2. Shorten / decouple the fallback so a bounded wait reliably proceeds even without a re-trigger (pair with the sweep), and/or anchor head-age so a no-op rebase doesn't reset the clock.
3. Config escape hatch: allow dropping a bot from the required advisory set (keep it advisory-only) via env, so an out-of-quota bot (Codex) can be removed without code change.

Separate (infra/billing — not code)

CodeRabbit and Codex have exhausted prepaid credits org-wide, so they aren't reviewing at all right now. Even with a perfect gate, restoring those credits (or accepting their absence via fix #1/#3) is needed for advisory coverage.

---

Filed from a triage of why pr-review wasn't approving green PRs. Evidence: run 27478909009 (#623) + rate-limit comments on the PRs listed above.

[don-petry]

Dev-Lead Implementation Plan

Issue: #657 — pr-review never auto-approves when advisory bots are rate-limited/out-of-quota

Scope

Implement recommended fix #1 (preferred): wire up the half-built RATE_LIMITED state in scripts/lib/advisory-review-gate.sh so an out-of-quota advisory bot that posts a usage-limit comment is detected, classified RATE_LIMITED, and treated as non-participating — dropping it from the required advisory set so it can no longer hold the gate open indefinitely.

Implementation Checklist

• scripts/lib/advisory-review-gate.sh — add known rate-limit/usage-limit marker regex (CodeRabbit "Review limit reached"/"used up its prepaid credits", Codex "reached your Codex usage limits")
• scripts/lib/advisory-review-gate.sh — get_advisory_bot_states: emit state: RATE_LIMITED for an advisory-bot comment whose body matches a marker
• scripts/lib/advisory-review-gate.sh — check_advisory_reviews: count RATE_LIMITED/UNSUPPORTED bots as responded and subtract them from the required total (effective_total, clamped to ≥1) so a permanently out-of-quota bot doesn't block approval
• Keep timeout fallbacks unchanged (head-age / quiescence)

Tests to Write (TDD, in tests/dev-lead/unit/test_advisory_review_gate.bats)

• Runtime: rate-limited Codex comment + 2 real reviews (copilot absent), recent head → gate returns 0 (regression: previously skipped/return 1)
• Classification: a Codex usage-limit comment is rendered as RATE_LIMITED in gate output
• Negative: a plain Codex comment (no marker) is NOT excluded — gate still waits (return 1) when a genuine bot is absent
• Structural: script references RATE_LIMITED detection + effective_total

Test Command

bats tests/dev-lead/unit/ (full repo lint: bash .dev-lead/scripts/dev-lead-lint.sh)

[don-petry]

Dev-Lead: Implementation Complete

Implemented recommended fix #1 (preferred) — wired up the half-built RATE_LIMITED state so an out-of-quota advisory bot can no longer hold the gate open.

Plan Execution

• Added RATE_LIMIT_MARKERS regex (CodeRabbit "Review limit reached"/"used up its prepaid credits"; Codex "reached your Codex usage limits")
• get_advisory_bot_states: a comment from an advisory bot whose body matches a marker is now classified state: RATE_LIMITED (case-insensitive) instead of COMMENTED
• check_advisory_reviews: RATE_LIMITED/UNSUPPORTED bots are treated as responded and subtracted from the required total (effective_total = total − unavailable, clamped to ≥1); the approval threshold now compares against effective_total
• Timeout fallbacks (head-age / quiescence) left unchanged; fallback log lines reworded to report the effective required count
• TDD: 5 new bats tests written before implementation

Behaviour

With Codex out of quota (posts its usage-limit comment) and Copilot absent, two genuine advisory reviews (Gemini + SonarCloud) now satisfy the gate immediately — effective_total drops from 4 to 3 and Codex counts as a response — so the PR is approved instead of skipping with waiting-for-advisory-bots. A plain (non-marker) comment is not excluded, so the gate still waits when a real reviewer is genuinely missing.

Test Results

bats tests/dev-lead/unit/      → 458 ok / 0 not ok
bats .../test_advisory_review_gate.bats → 44 ok (5 new, incl. red→green regression for out-of-quota bot)
bash .dev-lead/scripts/dev-lead-lint.sh → all checks passed
shellcheck --severity=warning -x scripts/lib/advisory-review-gate.sh → clean

Files Changed

• scripts/lib/advisory-review-gate.sh: rate-limit marker detection in get_advisory_bot_states; effective_total exclusion logic in check_advisory_reviews
• tests/dev-lead/unit/test_advisory_review_gate.bats: 5 new tests (marker constant, effective_total, RATE_LIMITED classification, out-of-quota-doesn't-block flip, no-over-exclusion guard)

Notes

• Scope limited to fix test issue from agent #1 as the issue instructed ("pick, keep minimal"). Fix Go-live improvements for PR review agent #2 (decouple fallback / anchor head-age) and fix Add @claude delegation, auto-merge, and rebase handling #3 (env escape hatch) were not implemented and remain open follow-ups if needed.
• A genuinely silent out-of-quota bot (posts nothing) still relies on the existing timeout fallbacks — self-healing only triggers once the bot posts its usage-limit comment, which is the documented behaviour across the cited PRs.
• The separate infra/billing item (restoring CodeRabbit/Codex prepaid credits) is out of code scope.