You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In run #715, the Copilot CLI exited with error: Invalid command format (a command syntax error), but is_rate_limited classified it as a rate limit and returned exit code 2, causing the session to abort:
[tier1] usage/rate limit detected — exiting with code 2 for engine fallback
limit message: (stderr: error: Invalid command format.
Did you mean: copilot -i "suggest -p ..."
The detector inspects stdout/stderr for keyword matches. Its current pattern set is too broad and matches on phrases like Invalid command format or similar output that has nothing to do with API rate limiting. The consequence is catastrophic: a simple CLI misconfiguration aborts the entire batch session.
Acceptance criteria
Audit current patterns: enumerate every regex/string pattern in is_rate_limited (and any callers) and justify each one against actual rate-limit response bodies from Claude, Gemini, and Copilot CLI.
Tighten patterns: remove or narrow patterns that can match non-rate-limit errors. At minimum, Invalid command format (and the Did you mean: suggestion) must NOT trigger a rate-limit exit.
Logging: when is_rate_limited returns false but the command failed, log the actual error category (e.g., [error] CLI format error — treating as per-PR failure) so post-mortem analysis is unambiguous.
No regression: existing rate-limit strings from Claude (overloaded, 529, rate_limit_error) and Copilot (rate limit, quota) still correctly trigger exit code 2.
Expected tests
Unit (bats): is_rate_limited returns false for each of the following inputs:
error: Invalid command format.
Did you mean: copilot -i ...
unknown flag: --foo
command not found
Unit (bats): is_rate_limited returns true for known rate-limit strings:
overloaded_error / 529
rate limit exceeded
quota exceeded
usage limit reached
Integration: a simulated CLI format-error (mocked exit 1 + Invalid command format stderr) results in a per-PR failure (exit 1) — not a session abort (exit 2) — and the next candidate PR is processed.
Parent issue
#145
Problem
In run #715, the Copilot CLI exited with
error: Invalid command format(a command syntax error), butis_rate_limitedclassified it as a rate limit and returned exit code 2, causing the session to abort:The detector inspects stdout/stderr for keyword matches. Its current pattern set is too broad and matches on phrases like
Invalid command formator similar output that has nothing to do with API rate limiting. The consequence is catastrophic: a simple CLI misconfiguration aborts the entire batch session.Acceptance criteria
is_rate_limited(and any callers) and justify each one against actual rate-limit response bodies from Claude, Gemini, and Copilot CLI.Invalid command format(and theDid you mean:suggestion) must NOT trigger a rate-limit exit.is_rate_limitedreturns false but the command failed, log the actual error category (e.g.,[error] CLI format error — treating as per-PR failure) so post-mortem analysis is unambiguous.overloaded,529,rate_limit_error) and Copilot (rate limit,quota) still correctly trigger exit code 2.Expected tests
is_rate_limitedreturns false for each of the following inputs:error: Invalid command format.Did you mean: copilot -i ...unknown flag: --foocommand not foundis_rate_limitedreturns true for known rate-limit strings:overloaded_error/529rate limit exceededquota exceededusage limit reachedInvalid command formatstderr) results in a per-PR failure (exit 1) — not a session abort (exit 2) — and the next candidate PR is processed.