perf: supervisor auth caching and auto model tier routing#794
perf: supervisor auth caching and auto model tier routing#794marcusquinn merged 1 commit intomainfrom
Conversation
Three efficiency improvements to reduce compute cost and API overhead: 1. Cached check_gh_auth() — cache auth validation result for 5 minutes instead of calling gh api user on every invocation. Saves ~288 API calls/day at 2-minute pulse intervals. Cache stored in supervisor dir with automatic invalidation on failure. 2. Task complexity classifier (classify_task_complexity) — pattern-match task descriptions to auto-route simple tasks (docs, config, renaming, typos, slash commands) to sonnet tier (~5x cheaper than opus). Complex tasks (architecture, security, novel features) stay on opus. Defaults to opus for safety — only downgrades when confident. 3. Wired classifier into resolve_task_model as priority 3 (after explicit model and subagent frontmatter, before default coding tier). Tasks like 'Update AGENTS.md progressive disclosure table' now dispatch on sonnet instead of opus, saving significant API cost per batch.
Summary of ChangesHello @marcusquinn, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request enhances the supervisor script by optimizing resource usage and cost efficiency. It introduces caching for GitHub authentication to minimize API calls and implements an intelligent system to classify task complexity, routing tasks to appropriate AI models based on their nature. This dynamic model selection aims to reduce operational costs by leveraging cheaper models for simpler tasks, without compromising performance for complex ones. Highlights
Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
WalkthroughA new task complexity classifier was introduced to intelligently route tasks between model tiers based on description patterns. Concurrently, GitHub authentication caching was enhanced to reduce API calls during cron operations. The classifier integrates into the model resolution flow as a prior step before fallback logic. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~22 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🔍 Code Quality Report�[0;35m[MONITOR]�[0m Code Review Monitoring Report �[0;34m[INFO]�[0m Latest Quality Status: �[0;34m[INFO]�[0m Recent monitoring activity: 📈 Current Quality Metrics
Generated on: Mon Feb 9 18:18:20 UTC 2026 Generated by AI DevOps Framework Code Review Monitoring |
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces two significant performance and cost-saving improvements: caching GitHub authentication status to reduce API calls, and a new task complexity classifier to route simpler tasks to cheaper AI models. The implementation is solid and the new features are well-documented. My review focuses on ensuring adherence to the repository's shell scripting style guide. I've identified a few areas where error handling and variable declarations can be aligned with the established conventions to improve script robustness and maintainability.
| mkdir -p "$(dirname "$cache_file")" 2>/dev/null || true | ||
| echo "ok" > "$cache_file" 2>/dev/null || true |
There was a problem hiding this comment.
The use of 2>/dev/null for blanket error suppression on these lines violates the repository style guide (line 50), which states that 2>/dev/null is only acceptable when redirecting to log files. While caching is a non-critical operation, suppressing errors from mkdir or echo can hide underlying issues like file permissions, making debugging more difficult. The || true guard is sufficient to prevent the script from exiting due to set -e. This feedback also applies to lines 271-272 and 275.
| mkdir -p "$(dirname "$cache_file")" 2>/dev/null || true | |
| echo "ok" > "$cache_file" 2>/dev/null || true | |
| mkdir -p "$(dirname "$cache_file")" || true | |
| echo "ok" > "$cache_file" || true |
References
- The style guide prohibits blanket error suppression with
2>/dev/nullunless output is being redirected to a log file. Errors should be visible unless explicitly handled. (link)
| # security, anything requiring deep reasoning. | ||
| ####################################### | ||
| classify_task_complexity() { | ||
| local description="$1" |
There was a problem hiding this comment.
This variable declaration violates the repository style guide (line 11), which requires declaring and assigning local variables on separate lines. This practice ensures that the exit code of the assignment is not masked, which is important for scripts running under set -e.
| local description="$1" | |
| local description | |
| description="$1" |
References
- The style guide mandates that
localvariable declaration and assignment should be separate statements to preserve the exit code of the assignment command. (link)
| # Route simple tasks (docs, config, renaming) to sonnet (~5x cheaper) | ||
| # Keep complex tasks (architecture, novel features) on opus | ||
| local task_desc | ||
| task_desc=$(db "$SUPERVISOR_DB" "SELECT description FROM tasks WHERE id = '$(sql_escape "$task_id")';" 2>/dev/null || echo "") |
There was a problem hiding this comment.
The use of 2>/dev/null here for blanket error suppression violates the repository style guide (line 50). The guide states that 2>/dev/null is only acceptable when redirecting to log files. Suppressing the error from the db command can hide underlying database issues, making debugging more difficult. The || echo "" guard is sufficient to handle command failure gracefully without exiting the script.
| task_desc=$(db "$SUPERVISOR_DB" "SELECT description FROM tasks WHERE id = '$(sql_escape "$task_id")';" 2>/dev/null || echo "") | |
| task_desc=$(db "$SUPERVISOR_DB" "SELECT description FROM tasks WHERE id = '$(sql_escape "$task_id")';" || echo "") |
References
- The style guide prohibits blanket error suppression with
2>/dev/nullunless output is being redirected to a log file. Errors from database commands should be visible to aid debugging. (link)
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
.agents/scripts/supervisor-helper.sh (1)
2889-2957: Prefer complex-pattern override to avoid under-tiering.Right now simple patterns short‑circuit before complex ones, so mixed descriptions (e.g., “update docs for migration”) can route to Sonnet even when a complex keyword is present. I’d recommend checking complex patterns first to keep the safe default.
♻️ Suggested ordering (complex wins)
- # Patterns that indicate simple/docs/config tasks → sonnet - local simple_patterns=( - "update.*readme" - "update.*docs" - "update.*documentation" - "add.*comment" - "add.*reference" - "update.*reference" - "rename" - "fix.*typo" - "update.*version" - "bump.*version" - "update.*changelog" - "add.*to.*index" - "update.*index" - "wire.*up.*command" - "add.*slash.*command" - "update.*agents\.md" - "progressive.*disclosure" - "cross-reference" - ) - - for pattern in "${simple_patterns[@]}"; do - if [[ "$desc_lower" =~ $pattern ]]; then - echo "sonnet" - return 0 - fi - done - - # Patterns that indicate complex tasks → opus - local complex_patterns=( - "architect" - "design.*system" - "security.*audit" - "refactor.*major" - "migration" - "novel" - "from.*scratch" - "implement.*new.*system" - "multi.*provider" - "cross.*model" - "quality.*gate" - "fallback.*chain" - ) - - for pattern in "${complex_patterns[@]}"; do - if [[ "$desc_lower" =~ $pattern ]]; then - echo "opus" - return 0 - fi - done + # Patterns that indicate complex tasks → opus + local complex_patterns=( + "architect" + "design.*system" + "security.*audit" + "refactor.*major" + "migration" + "novel" + "from.*scratch" + "implement.*new.*system" + "multi.*provider" + "cross.*model" + "quality.*gate" + "fallback.*chain" + ) + + for pattern in "${complex_patterns[@]}"; do + if [[ "$desc_lower" =~ $pattern ]]; then + echo "opus" + return 0 + fi + done + + # Patterns that indicate simple/docs/config tasks → sonnet + local simple_patterns=( + "update.*readme" + "update.*docs" + "update.*documentation" + "add.*comment" + "add.*reference" + "update.*reference" + "rename" + "fix.*typo" + "update.*version" + "bump.*version" + "update.*changelog" + "add.*to.*index" + "update.*index" + "wire.*up.*command" + "add.*slash.*command" + "update.*agents\.md" + "progressive.*disclosure" + "cross-reference" + ) + + for pattern in "${simple_patterns[@]}"; do + if [[ "$desc_lower" =~ $pattern ]]; then + echo "sonnet" + return 0 + fi + done
| # Cache auth check result for 5 minutes to avoid repeated API calls. | ||
| # Each pulse calls this 1-5 times; cron runs every 2-5 minutes. | ||
| # Caching saves ~288 API calls/day at 2-min pulse intervals. | ||
| local cache_file="${SUPERVISOR_DIR:-.}/.gh-auth-cache" | ||
| local cache_ttl=300 # 5 minutes | ||
|
|
||
| if [[ -f "$cache_file" ]]; then | ||
| local cache_age | ||
| local cache_mtime | ||
| cache_mtime=$(stat -f '%m' "$cache_file" 2>/dev/null || stat -c '%Y' "$cache_file" 2>/dev/null || echo "0") | ||
| cache_age=$(( $(date +%s) - cache_mtime )) | ||
| if [[ "$cache_age" -lt "$cache_ttl" ]]; then | ||
| local cached_result | ||
| cached_result=$(cat "$cache_file" 2>/dev/null || echo "") | ||
| if [[ "$cached_result" == "ok" ]]; then | ||
| return 0 | ||
| fi | ||
| # Cached failure — still retry in case token was refreshed | ||
| fi | ||
| fi | ||
|
|
||
| # If GH_TOKEN is set, verify it works with a lightweight API call | ||
| if [[ -n "${GH_TOKEN:-}" ]]; then | ||
| gh api user --jq '.login' >/dev/null 2>&1 && return 0 | ||
| if gh api user --jq '.login' >/dev/null 2>&1; then | ||
| mkdir -p "$(dirname "$cache_file")" 2>/dev/null || true | ||
| echo "ok" > "$cache_file" 2>/dev/null || true | ||
| return 0 | ||
| fi | ||
| fi | ||
| # Fall back to gh auth status (works interactively with keyring) | ||
| gh auth status >/dev/null 2>&1 && return 0 | ||
| if gh auth status >/dev/null 2>&1; then | ||
| mkdir -p "$(dirname "$cache_file")" 2>/dev/null || true | ||
| echo "ok" > "$cache_file" 2>/dev/null || true | ||
| return 0 | ||
| fi | ||
| echo "fail" > "$cache_file" 2>/dev/null || true |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, let's see if the file exists and check its size
fd -p "supervisor-helper.sh" .agents/scripts/Repository: marcusquinn/aidevops
Length of output: 100
🏁 Script executed:
# Run ShellCheck on the supervisor-helper.sh file
shellcheck .agents/scripts/supervisor-helper.sh 2>&1 || trueRepository: marcusquinn/aidevops
Length of output: 2027
🏁 Script executed:
# Let's examine the specific lines mentioned in the review (240-275)
sed -n '240,275p' .agents/scripts/supervisor-helper.shRepository: marcusquinn/aidevops
Length of output: 1595
🏁 Script executed:
# Check the other segments mentioned in the scratchpad (2889-2957 and 2962-3027)
wc -l .agents/scripts/supervisor-helper.sh
sed -n '2889,2957p' .agents/scripts/supervisor-helper.sh
sed -n '2962,3027p' .agents/scripts/supervisor-helper.shRepository: marcusquinn/aidevops
Length of output: 5017
ShellCheck violations found; zero-violation requirement not met.
The auth-check caching logic (lines 240-275) is solid and follows guidelines well. However, ShellCheck reports violations across the file that prevent zero-compliance:
- SC2034 (line 8372, 9631): Unused variables
duration_info,c_magenta,c_bg_black— remove or export - SC2015 (line 9984):
A && B || Cpattern is not true if-then-else — refactor to proper conditional - SC2016 (line 5151): Single quotes with variable expansion — use double quotes
- SC2329 (line 7609): Function
process_verify_queueis never invoked — remove or add usage - SC1091 (line 161): Source file not followed — run shellcheck with
-xflag or verify sourcing
Per coding guidelines: "Run ShellCheck with zero violations on all scripts in .agents/scripts/" — this file must resolve all violations before merge.
1 participant
Summary
check_gh_auth()results for 5 minutes to eliminate redundant GitHub API calls (~288/day saved)resolve_task_modelas step 3 in the resolution chainChanges
1. Cached
check_gh_auth()(lines 239-278)Before: Every call to
check_gh_auth()hitgh api userorgh auth status. Called 1-5 times per pulse, pulses every 2-5 minutes = ~288-720 API calls/day.After: Result cached to
$SUPERVISOR_DIR/.gh-auth-cachewith 5-minute TTL. Subsequent calls within the window return instantly from cache. Cache invalidates on failure so token refreshes are picked up.2.
classify_task_complexity()(new function)Pattern-matches task descriptions against known simple/complex indicators:
Tested against real task descriptions from backlog-09:
3. Wired into
resolve_task_model()(step 3)Resolution priority is now:
model:fieldCost Impact
Based on backlog-09 (20 tasks):
Testing
bash -n: syntax OKSummary by CodeRabbit
Release Notes
New Features
Improvements