feat: support multiple AI code reviewers in PR loop#295
Conversation
Add regex-based detection for CodeRabbit, Gemini Code Assist, Augment Code, and GitHub Copilot review comments. Previously only CodeRabbit was monitored, causing /pr-loop to miss feedback from other AI reviewers. Changes: - Add AI_REVIEWERS constant with regex pattern - Update get_pr_feedback() to detect all AI reviewer comments - Update check_and_trigger_review() to check all AI reviewer times - Add check_unresolved_review_comments() for detecting pending reviews - Document supported AI reviewers in workflows/pr.md Closes #294
|
Warning Rate limit exceeded
⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. WalkthroughThe PR extends the PR review loop to monitor multiple AI code reviewers (CodeRabbit, Gemini Code Assist, Augment Code, GitHub Copilot). It adds an Changes
Sequence Diagram(s)sequenceDiagram
participant PRLoop as PR Loop
participant GH as GitHub API
participant Unresolved as Unresolved Check
participant Approver as Approver/PR Status
PRLoop->>GH: Query PR comments/reviews using AI_REVIEWERS filter
GH-->>PRLoop: Return AI reviewer comments & last review timestamp
PRLoop->>Unresolved: check_unresolved_review_comments(pr_number)
Unresolved->>GH: Query unresolved review comments for AI reviewers
GH-->>Unresolved: Unresolved count/details
alt Unresolved comments exist
Unresolved-->>PRLoop: Return unresolved status
PRLoop->>PRLoop: Fetch & display AI feedback, block approval
else No unresolved comments
Unresolved-->>PRLoop: Return clean status
PRLoop->>Approver: Proceed with approval/READY actions
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary of ChangesHello @alex-solovyev, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the Highlights
Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
🔍 Code Quality Report�[0;35m[MONITOR]�[0m Code Review Monitoring Report �[0;34m[INFO]�[0m Latest Quality Status: �[0;34m[INFO]�[0m Recent monitoring activity: 📈 Current Quality Metrics
Generated on: Tue Feb 3 05:02:56 UTC 2026 Generated by AI DevOps Framework Code Review Monitoring |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request is a great enhancement, adding support for multiple AI code reviewers to the PR loop. The changes are well-structured, using a constant for the reviewer patterns and updating the relevant functions. However, I've identified a few areas for improvement. The regex for matching bot usernames is a bit too broad and could lead to false positives. More critically, the script currently suppresses API errors, which could cause the loop to incorrectly determine that a PR is ready to merge when there are unresolved comments. I've provided suggestions to make the regex more specific and to add explicit error handling for API calls. I also included a minor suggestion to improve the documentation.
| unresolved_count=$(gh api "repos/{owner}/{repo}/pulls/${pr_number}/comments" \ | ||
| --jq --arg bots "$AI_REVIEWERS" \ | ||
| '[.[] | select(.user.login | test($bots; "i")) | select(.in_reply_to_id == null)] | length' \ | ||
| 2>/dev/null || echo "0") | ||
|
|
||
| if [[ "$unresolved_count" -gt 0 ]]; then | ||
| print_warning "Found $unresolved_count unresolved AI review comments" | ||
| return 1 | ||
| fi | ||
| return 0 |
There was a problem hiding this comment.
This command suppresses errors from the gh api call using 2>/dev/null and provides a default value of 0 using || echo "0". This is dangerous because if the API call fails for any reason (e.g., network issues, expired token), unresolved_count will become 0, and the function will incorrectly report that there are no unresolved comments. This could lead to a PR being merged while it still has pending AI feedback. The script should instead fail explicitly if the API call does not succeed and return a valid number.
| unresolved_count=$(gh api "repos/{owner}/{repo}/pulls/${pr_number}/comments" \ | |
| --jq --arg bots "$AI_REVIEWERS" \ | |
| '[.[] | select(.user.login | test($bots; "i")) | select(.in_reply_to_id == null)] | length' \ | |
| 2>/dev/null || echo "0") | |
| if [[ "$unresolved_count" -gt 0 ]]; then | |
| print_warning "Found $unresolved_count unresolved AI review comments" | |
| return 1 | |
| fi | |
| return 0 | |
| unresolved_count=$(gh api "repos/{owner}/{repo}/pulls/${pr_number}/comments" \ | |
| --jq --arg bots "$AI_REVIEWERS" \ | |
| '[.[] | select(.user.login | test($bots; "i")) | select(.in_reply_to_id == null)] | length' 2>/dev/null) | |
| if ! [[ "$unresolved_count" =~ ^[0-9]+$ ]]; then | |
| print_error "Error: Failed to retrieve unresolved comment count from GitHub API. Cannot proceed safely." | |
| return 1 | |
| fi | |
| if [[ "$unresolved_count" -gt 0 ]]; then | |
| print_warning "Found $unresolved_count unresolved AI review comments" | |
| return 1 | |
| fi | |
| return 0 |
|
|
||
| # AI code reviewers (regex pattern for jq test()) | ||
| # Supported: CodeRabbit, Gemini Code Assist, Augment Code, GitHub Copilot | ||
| readonly AI_REVIEWERS="coderabbit|gemini-code-assist|augment-code|copilot" |
There was a problem hiding this comment.
The current regex pattern for AI_REVIEWERS is too broad. It performs a substring match because jq's test() function checks for containment when no anchors are used. This could incorrectly match usernames that simply contain the names of the bots (e.g., a user named john-copilot). This could lead to unintended behavior. The regex should be made more specific by using anchors (^, $) and escaping special characters to ensure only the intended bot usernames are matched.
| readonly AI_REVIEWERS="coderabbit|gemini-code-assist|augment-code|copilot" | |
| readonly AI_REVIEWERS="^coderabbit|^gemini-code-assist\\[bot\\]$|^augment-code\\[bot\\]$|^copilot\\[bot\\]$" |
| ai_review_comments=$(gh api "repos/{owner}/{repo}/pulls/${pr_number}/comments" \ | ||
| --jq --arg bots "$AI_REVIEWERS" \ | ||
| '.[] | select(.user.login | test($bots; "i")) | "\(.user.login): \(.body)"' \ | ||
| 2>/dev/null | head -20 || echo "") |
There was a problem hiding this comment.
The use of 2>/dev/null || echo "" suppresses any errors from the gh api call. If the API fails (e.g., due to network issues or an invalid token), the script will silently continue as if there were no comments, and no feedback will be displayed. While the consequence here is less severe than in other parts of the script, it's good practice to handle API errors explicitly to make debugging easier. Consider logging a warning if the API call fails.
.agent/workflows/pr.md
Outdated
| |----------|---------------------|---------| | ||
| | CodeRabbit | `coderabbit*` | AI-powered code review with suggestions | | ||
| | Gemini Code Assist | `gemini-code-assist[bot]` | Google's AI code review | | ||
| | Augment Code | `augment-code[bot]` | Augment AI code review | |
There was a problem hiding this comment.
The description for 'Augment Code' in the 'Purpose' column is tautological ('Augment AI code review'). A more descriptive explanation would be more helpful for readers to understand its function.
| | Augment Code | `augment-code[bot]` | Augment AI code review | | |
| | Augment Code | `augment-code[bot]` | AI-powered code review and improvement | |
🤖 Augment PR SummarySummary: Expands the PR loop so it can surface feedback from multiple AI code review bots (not just CodeRabbit), reducing false “Ready to Merge” outcomes. Changes:
🤖 Was this summary useful? React with 👍 or 👎 |
![augmentcode[bot]](https://avatars.githubusercontent.com/in/1027498?s=60&v=4){kind=small}
| # Get AI reviewer comments (CodeRabbit, Gemini Code Assist, Augment Code, Copilot) | ||
| local ai_review_comments | ||
| ai_review_comments=$(gh api "repos/{owner}/{repo}/pulls/${pr_number}/comments" \ | ||
| --jq --arg bots "$AI_REVIEWERS" \ |
There was a problem hiding this comment.
gh api supports --jq, but it doesn’t accept additional jq flags like --arg; this will likely fail with an “unknown flag” error and (because of || echo "") silently drop AI reviewer feedback.
Severity: high
Other Locations
.agent/scripts/quality-loop-helper.sh:698.agent/scripts/quality-loop-helper.sh:748
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
| # Arguments: $1 - PR number | ||
| # Returns: 0 if no unresolved comments, 1 if unresolved comments exist | ||
| # Output: Warning message if unresolved comments found | ||
| check_unresolved_review_comments() { |
There was a problem hiding this comment.
check_unresolved_review_comments is defined but not called anywhere in the PR loop, so it won’t currently prevent /pr-loop from reporting success even when AI review comments exist.
Severity: medium
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
| local unresolved_count | ||
| unresolved_count=$(gh api "repos/{owner}/{repo}/pulls/${pr_number}/comments" \ | ||
| --jq --arg bots "$AI_REVIEWERS" \ | ||
| '[.[] | select(.user.login | test($bots; "i")) | select(.in_reply_to_id == null)] | length' \ |
There was a problem hiding this comment.
select(.in_reply_to_id == null) counts top-level review comments, not “unresolved” threads, so this may warn even after comments have been addressed/resolved in the GitHub UI.
Severity: medium
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
- Fix gh api --jq not supporting --arg flag by piping to jq - Make AI_REVIEWERS regex more specific with anchors to prevent false positives - Add explicit error handling for GitHub API calls - Call check_unresolved_review_comments() in READY state before declaring merge-ready - Improve Augment Code description in pr.md documentation
🔍 Code Quality Report�[0;35m[MONITOR]�[0m Code Review Monitoring Report �[0;34m[INFO]�[0m Latest Quality Status: �[0;34m[INFO]�[0m Recent monitoring activity: 📈 Current Quality Metrics
Generated on: Tue Feb 3 05:11:16 UTC 2026 Generated by AI DevOps Framework Code Review Monitoring |
|
augment review |
| if [[ -z "$api_response" ]]; then | ||
| print_warning "Failed to fetch PR comments from GitHub API" | ||
| else | ||
| ai_review_comments=$(echo "$api_response" | jq -r --arg bots "$AI_REVIEWERS" \ |
There was a problem hiding this comment.
api_response contains raw JSON; piping it through echo can mangle backslash escapes on some shells and lead to intermittent jq parse failures (which would make AI feedback appear “missing”). This same pattern appears in the review-time and unresolved-count parsing too.
Severity: medium
Other Locations
.agent/scripts/quality-loop-helper.sh:705.agent/scripts/quality-loop-helper.sh:764
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
| echo "<promise>PR_APPROVED</promise>" | ||
| return 0 | ||
| # Check for unresolved AI review comments before declaring ready | ||
| if ! check_unresolved_review_comments "$pr_number"; then |
There was a problem hiding this comment.
if ! check_unresolved_review_comments ... treats both “unresolved comments” (exit 1) and “API error” (exit 2) the same, so transient GitHub/gh failures will be reported as unresolved feedback and can prevent ever reaching the READY path. If you intend API errors to be a hard-stop, it may be worth distinguishing the 2 case explicitly.
Severity: medium
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
.agent/workflows/pr.md
Outdated
| |----------|---------------------|---------| | ||
| | CodeRabbit | `coderabbit*` | AI-powered code review with suggestions | | ||
| | Gemini Code Assist | `gemini-code-assist[bot]` | Google's AI code review | | ||
| | Augment Code | `augment-code[bot]` | AI-powered code review and improvement | |
There was a problem hiding this comment.
Docs list Augment as augment-code[bot], but the implementation also matches augmentcode[bot] (no dash). It’d be easy for users to get confused about which bot accounts are actually supported if the table doesn’t reflect both.
Severity: low
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In @.agent/scripts/quality-loop-helper.sh:
- Around line 749-777: The function check_unresolved_review_comments currently
fetches PR comments via the REST endpoint and uses in_reply_to_id to infer
unresolved status; replace that REST call with a GitHub GraphQL query (using gh
api graphql) to fetch the pullRequest.reviewThreads.nodes and use each
node.isResolved to count unresolved threads for the given PR number. Update the
api_response handling in check_unresolved_review_comments to run a GraphQL query
for repository(owner:, name:) { pullRequest(number: $pr_number) {
reviewThreads(first: 100) { nodes { isResolved author { login } } } } }, filter
threads where isResolved == false and author.login matches entries in
$AI_REVIEWERS (case-insensitive), parse the count into unresolved_count,
preserve the same error checks and return codes (0/1/2) and warning message when
unresolved_count > 0. Ensure JSON parsing uses jq safely and retain existing
failure messages from print_error/print_warning.
🧹 Nitpick comments (2)
.agent/scripts/quality-loop-helper.sh (1)
647-661: Paginate AI comment fetch to avoid missing feedback.
gh apidefaults to the first page; if a PR has >30 review comments, AI feedback can be missed and the loop may incorrectly proceed. Consider--paginatehere.♻️ Suggested change
- api_response=$(gh api "repos/{owner}/{repo}/pulls/${pr_number}/comments" 2>/dev/null) + api_response=$(gh api "repos/{owner}/{repo}/pulls/${pr_number}/comments" --paginate 2>/dev/null).agent/workflows/pr.md (1)
100-112: Prefer a file:line reference to avoid doc drift.
This table mirrors script implementation and can stale; consider pointing tofile:.agent/scripts/quality-loop-helper.sh:59-61and a short sentence instead of inline details.As per coding guidelines: “.agent/**/*.md: Apply progressive disclosure… use
file:linereferences to point to actual implementation instead of inline code snippets.”
- Use printf instead of echo for JSON to prevent backslash escape issues - Distinguish API errors (exit 2) from unresolved comments (exit 1) in READY handler - Document both augment-code[bot] and augmentcode[bot] in pr.md
🔍 Code Quality Report�[0;35m[MONITOR]�[0m Code Review Monitoring Report �[0;34m[INFO]�[0m Latest Quality Status: �[0;34m[INFO]�[0m Recent monitoring activity: 📈 Current Quality Metrics
Generated on: Tue Feb 3 05:18:58 UTC 2026 Generated by AI DevOps Framework Code Review Monitoring |
Replace REST in_reply_to_id check with GraphQL reviewThreads query. The REST field only indicates if a comment is top-level, not whether the thread is resolved. GraphQL isResolved field correctly reflects the resolution status as shown in GitHub UI.
🔍 Code Quality Report�[0;35m[MONITOR]�[0m Code Review Monitoring Report �[0;34m[INFO]�[0m Latest Quality Status: �[0;34m[INFO]�[0m Recent monitoring activity: 📈 Current Quality Metrics
Generated on: Tue Feb 3 05:22:44 UTC 2026 Generated by AI DevOps Framework Code Review Monitoring |
SC2016 flags $owner, $repo, $number as unexpanded shell variables, but they are GraphQL variables passed via -f/-F flags, not shell vars. Single quotes are intentional to pass the literal query string.
🔍 Code Quality Report�[0;35m[MONITOR]�[0m Code Review Monitoring Report �[0;34m[INFO]�[0m Latest Quality Status: �[0;34m[INFO]�[0m Recent monitoring activity: 📈 Current Quality Metrics
Generated on: Tue Feb 3 05:38:43 UTC 2026 Generated by AI DevOps Framework Code Review Monitoring |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In @.agent/scripts/quality-loop-helper.sh:
- Around line 749-793: The check_unresolved_review_comments function currently
counts all unresolved review threads but should only consider AI reviewer
threads; update the GraphQL selection to include comments' author login (add
comments { nodes { author { login } } } to the pullRequest.reviewThreads query)
and change the jq extraction (unresolved_count) to filter unresolved threads
where the first comment author login matches the AI reviewer pattern variable
($AI_REVIEWERS) using jq's test() (same pattern used elsewhere in this file),
ensuring unresolved_count uses that filtered length; keep references to
check_unresolved_review_comments, api_response, and $AI_REVIEWERS when making
the change.
| # Check for unresolved review threads on a PR using GraphQL | ||
| # Arguments: $1 - PR number | ||
| # Returns: 0 if no unresolved threads, 1 if unresolved threads exist, 2 on API error | ||
| # Output: Warning message if unresolved threads found | ||
| check_unresolved_review_comments() { | ||
| local pr_number="$1" | ||
|
|
||
| local repo_owner repo_name api_response unresolved_count | ||
| repo_owner=$(gh repo view --json owner -q '.owner.login' 2>/dev/null || echo "") | ||
| repo_name=$(gh repo view --json name -q '.name' 2>/dev/null || echo "") | ||
|
|
||
| if [[ -z "$repo_owner" || -z "$repo_name" ]]; then | ||
| print_error "Failed to resolve repo owner/name - cannot verify review status" | ||
| return 2 | ||
| fi | ||
|
|
||
| # shellcheck disable=SC2016 # GraphQL variables, not shell - single quotes intentional | ||
| api_response=$(gh api graphql -f query=' | ||
| query($owner:String!, $repo:String!, $number:Int!) { | ||
| repository(owner:$owner, name:$repo) { | ||
| pullRequest(number:$number) { | ||
| reviewThreads(first:100) { nodes { isResolved } } | ||
| } | ||
| } | ||
| }' -f owner="$repo_owner" -f repo="$repo_name" -F number="$pr_number" 2>/dev/null) | ||
|
|
||
| if [[ -z "$api_response" ]]; then | ||
| print_error "Failed to fetch PR review threads from GitHub API - cannot verify review status" | ||
| return 2 | ||
| fi | ||
|
|
||
| unresolved_count=$(printf '%s' "$api_response" | jq -r \ | ||
| '[.data.repository.pullRequest.reviewThreads.nodes[] | select(.isResolved == false)] | length' 2>/dev/null) | ||
|
|
||
| if ! [[ "$unresolved_count" =~ ^[0-9]+$ ]]; then | ||
| print_error "Failed to parse unresolved thread count - cannot proceed safely" | ||
| return 2 | ||
| fi | ||
|
|
||
| if [[ "$unresolved_count" -gt 0 ]]; then | ||
| print_warning "Found $unresolved_count unresolved review threads" | ||
| return 1 | ||
| fi | ||
| return 0 | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# Get broader context around the function
head -n 800 .agent/scripts/quality-loop-helper.sh | tail -n +730 | cat -n
# Search for references to AI_REVIEWERS in the file
rg -n "AI_REVIEWERS" .agent/scripts/quality-loop-helper.sh -B 2 -A 2
# Search for any comments about unresolved review filtering
rg -n "unresolved" .agent/scripts/quality-loop-helper.sh -B 3 -A 3Repository: marcusquinn/aidevops
Length of output: 6409
Filter unresolved threads by AI reviewer author to match stated intent.
The function checks for unresolved review threads from any source, but the calling context (line 869) and comment state "Check for unresolved AI review comments." Other functions in this file filter by $AI_REVIEWERS using jq's test() function—apply the same pattern here.
Update the GraphQL query to include comments { nodes { author { login } } } and filter the jq selection to exclude non-AI reviewers, matching the behavior at lines 654-655 and 705-706:
[.data.repository.pullRequest.reviewThreads.nodes[] | select(.isResolved == false and (.comments.nodes[0].author.login | test($bots; "i")))] | length
This ensures the automation only blocks on AI reviewer feedback, maintaining the intended gate before marking PRs as ready.
🤖 Prompt for AI Agents
In @.agent/scripts/quality-loop-helper.sh around lines 749 - 793, The
check_unresolved_review_comments function currently counts all unresolved review
threads but should only consider AI reviewer threads; update the GraphQL
selection to include comments' author login (add comments { nodes { author {
login } } } to the pullRequest.reviewThreads query) and change the jq extraction
(unresolved_count) to filter unresolved threads where the first comment author
login matches the AI reviewer pattern variable ($AI_REVIEWERS) using jq's test()
(same pattern used elsewhere in this file), ensuring unresolved_count uses that
filtered length; keep references to check_unresolved_review_comments,
api_response, and $AI_REVIEWERS when making the change.
Codacy and similar tools return ACTION_REQUIRED when issues are found, not FAILURE. Previously this was ignored, allowing PR to be marked READY while quality issues remained. Now ACTION_REQUIRED blocks like FAILURE.
|
🔍 Code Quality Report�[0;35m[MONITOR]�[0m Code Review Monitoring Report �[0;34m[INFO]�[0m Latest Quality Status: �[0;34m[INFO]�[0m Recent monitoring activity: 📈 Current Quality Metrics
Generated on: Tue Feb 3 05:44:52 UTC 2026 Generated by AI DevOps Framework Code Review Monitoring |
1 participant
Summary
Problem
quality-loop-helper.shonly monitored CodeRabbit reviews, ignoring feedback from other AI code review services. This caused/pr-loopto miss review comments and declare "Ready to Merge" prematurely.Evidence: In a managed project PR #2, Gemini Code Assist left comments that were not detected by the PR loop.
Changes
quality-loop-helper.shAI_REVIEWERSconstant with regex patternquality-loop-helper.shget_pr_feedback()to detect all AI reviewer commentsquality-loop-helper.shcheck_and_trigger_review()to check all AI reviewer timesquality-loop-helper.shcheck_unresolved_review_comments()functionworkflows/pr.mdSupported AI Reviewers
coderabbit*gemini-code-assist[bot]augment-code[bot]copilot[bot]Closes #294
Summary by CodeRabbit
New Features
Documentation