fix: replace gh copilot suggest with GitHub Models REST API - #151
Conversation
The `gh copilot suggest -p "$(cat <file>)"` invocation failed with "Invalid command format" because: 1. The `-p` flag is not valid syntax in modern `gh copilot` built-in versions. 2. `gh copilot suggest` is a shell-command suggestion tool; it does not accept large PR prompts or return structured JSON responses. 3. Passing a multi-thousand-line prompt via `$(cat ...)` can hit ARG_MAX. The non-zero exit was then misclassified by the rate-limit detector, which aborted the entire session and skipped all remaining PRs in the queue. Fix: replace all three copilot invocations (run_triage, run_agentic, run_duck) with a new `copilot_chat` helper that calls the GitHub Models REST API directly via curl. The API is OpenAI-compatible, versioned via `X-GitHub-Api-Version`, stable against gh CLI version changes, and accepts arbitrary prompt sizes using python3 for safe JSON encoding. Additional changes: - Add `COPILOT_API_MODEL` var (default `openai/o4-mini`) overrideable via env. - run_agentic/run_duck copilot cases now also write output to $OUTPUT_FILE so callers that check that path directly (deep review, audit) find the JSON. - Rate-limit responses (HTTP 429) are echoed to stdout so the existing `is_rate_limited()` detector fires correctly for engine fallback. - Add pre-flight smoke test in review-batch.sh: tests GitHub Models API connectivity with a one-liner prompt before processing any PRs, so auth or model errors surface immediately as a clear setup failure. - Add tests/test_copilot_chat.sh: unit tests for the JSON payload builder with edge-case prompts (quotes, newlines, # headings, large diffs, Unicode). Closes #147 Co-authored-by: Don Petry <don-petry@users.noreply.github.com>
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (3)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Move the `export COPILOT_API_MODEL` from the common exports block into the
`copilot)` case where the variable is set, so it is not exported as an
empty/unset variable when the engine is `claude` or `gemini`. The
`copilot_chat` fallback `${COPILOT_API_MODEL:-openai/o4-mini}` still works
correctly when `DUCK_ENGINE=copilot` under a non-copilot primary engine.
Co-authored-by: Don Petry <don-petry@users.noreply.github.com>
|
This PR is ready for review. @petry-projects/org-leads — please review and merge when approved. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request replaces the broken gh copilot suggest command with a direct integration to the GitHub Models REST API via a new copilot_chat function in scripts/engine.sh. It also introduces a pre-flight connectivity check in scripts/review-batch.sh and a new test suite for JSON payload generation. Review feedback identified a potential ARG_MAX limitation when handling large PR diffs as command-line arguments and recommended using temporary files for the API payload. Furthermore, the reviewer pointed out a likely typo in the default model name and requested improved rate-limit detection to support engine fallback mechanisms.
There was a problem hiding this comment.
Pull request overview
This PR updates the Copilot engine implementation to stop using gh copilot suggest (which is unstable across CLI versions and unsuitable for large/structured prompts) and instead call the GitHub Models REST API directly, improving reliability of the automated PR review pipeline.
Changes:
- Add a
copilot_chathelper inscripts/engine.shthat builds JSON safely viapython3and callshttps://models.github.ai/inference/chat/completionsviacurl. - Add a Copilot pre-flight smoke test in
scripts/review-batch.shto fail fast on auth/connectivity/model issues. - Add network-free unit tests for the JSON payload builder (
tests/test_copilot_chat.sh).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| scripts/engine.sh | Introduces copilot_chat, switches triage/deep/duck Copilot paths to GitHub Models REST API, adds COPILOT_API_MODEL. |
| scripts/review-batch.sh | Adds a Copilot-only pre-flight smoke test against GitHub Models API before reviewing PRs. |
| tests/test_copilot_chat.sh | Adds shell-based unit tests validating JSON encoding for edge-case prompts. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…e, streaming output - copilot_chat: write JSON body to mktemp file, pass to curl as @file to avoid ARG_MAX for large PR diffs (was --data-binary "$body") - copilot_chat: add :? guard on COPILOT_GITHUB_TOKEN for a clear error instead of generic "unbound variable" under set -u - run_agentic / run_duck copilot paths: stream directly to stdout (and tee to OUTPUT_FILE when set) rather than buffering the full response into a shell variable, which forced large outputs into memory and stripped trailing newlines - review-batch.sh pre-flight: fail fast if source engine.sh fails (was silently ignored with || true) - review-batch.sh pre-flight: build smoke-test JSON payload via python3 + temp file instead of shell string interpolation to avoid JSON injection if COPILOT_API_MODEL contains special characters - Clarify that openai/o4-mini is the correct April-2025 o4-generation model name, not a typo for o1-mini or gpt-4o-mini Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Summary
scripts/engine.shinvokedgh copilot suggest -p "$(cat <file>)"in three places. The-pflag is invalid in moderngh copilotbuilt-in versions, producingerror: Invalid command format. Additionally,gh copilot suggestis a shell-command suggestion tool — it cannot handle multi-thousand-line PR prompts or return structured JSON. The resulting non-zero exit was misclassified by the rate-limit detector, aborting the entire review session.run_triage,run_agentic,run_duck) with a newcopilot_chathelper that calls the GitHub Models REST API (https://models.github.ai/inference/chat/completions) viacurl. The API is OpenAI-compatible, versioned viaX-GitHub-Api-Version, and stable againstghCLI version changes.python3for JSON payload building — safely handles special characters,#headings, newlines, quotes, and large PR diffs without ARG_MAX issues.is_rate_limited()detector fires correctly for engine fallback (exit 2).review-batch.sh— verifies GitHub Models API connectivity before processing any PRs; format/auth errors now surface as a clear setup failure rather than mid-run false-positive rate-limits.tests/test_copilot_chat.shtests the JSON payload builder with edge-case prompts (quotes, newlines,#headings, large diffs, Unicode, backslashes).COPILOT_API_MODELenv var (defaultopenai/o4-mini) overrideable at the job level if the default model is unavailable.Changes
scripts/engine.shcopilot_chathelper; replace 3xgh copilot suggestcalls; addCOPILOT_API_MODELscripts/review-batch.shtests/test_copilot_chat.shTest plan
tests/test_copilot_chat.sh): payload builder handles all edge casesbash scripts/review-batch.shwithREVIEW_ENGINE=copilotagainst a real low-risk PR, confirm triage tier completes with exit 0Closes #147
Generated with Claude Code