Skip to content

Harden triage tier against prompt regressions and timeouts - #22

Merged
don-petry merged 1 commit into
mainfrom
claude/charming-hamilton-c15000
Apr 30, 2026
Merged

Harden triage tier against prompt regressions and timeouts#22
don-petry merged 1 commit into
mainfrom
claude/charming-hamilton-c15000

Conversation

@don-petry

Copy link
Copy Markdown
Collaborator

Summary

Two recent failures in the PR Review Agent workflow each aborted the hourly batch session-fatally — see the investigation report for full analysis.

This PR addresses both root causes with minimal, low-risk changes.

Changes

  • Bump TRIAGE_TIMEOUT_SEC default 180s → 300s (scripts/engine.sh). The 180s cap was tight for large PRs and offered no headroom; 300s matches ACTION_TIMEOUT_SEC and reduces the chance of a kill being misclassified as a deterministic failure (kills at 124 are retried; exit 1 is not).
  • Tighten the triage prompt (prompts/triage.md): explicitly forbid clarifying questions, plan proposals, and prose. Add a fallback rule so even on incomplete context the model must still emit JSON (with escalate=true and a signal explaining what was missing) rather than respond conversationally.

Test plan

  • Hourly cron run completes without aborting on the same PR set
  • Verify TRIAGE_TIMEOUT_SEC=300 shows in the workflow env block
  • Spot-check a triage log to confirm prompt update was picked up
  • Watch the next 24h of runs for any non-JSON failures

🤖 Generated with Claude Code

Two recent failures in the PR Review Agent workflow each aborted the
hourly batch session-fatally:

- Run #461: triage model returned conversational text asking
  clarifying questions instead of JSON, breaking the contract.
- Run #470: triage process exited 1 on a single PR (likely timeout
  or subprocess issue with no diagnostic output).

Changes:
1. Bump TRIAGE_TIMEOUT_SEC default 180s → 300s. The 180s cap was
   tight for large PRs and offers no headroom for the model to
   complete; 300s matches ACTION_TIMEOUT_SEC and reduces the chance
   of a kill being misclassified as a deterministic failure.
2. Tighten the triage prompt: explicitly forbid clarifying questions,
   plan proposals, and prose. Add a fallback rule that says even on
   incomplete context the model must still emit JSON (with
   escalate=true) rather than respond conversationally.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings April 30, 2026 02:24
@don-petry
don-petry merged commit 9019689 into main Apr 30, 2026
4 checks passed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens the tier-1 triage path in the PR review workflow to reduce session-fatal failures from model regressions (non-JSON output) and overly tight timeouts.

Changes:

  • Increase the default triage timeout from 180s to 300s.
  • Tighten the triage prompt to forbid clarifying questions/plans/prose and to require JSON output even when context is incomplete.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
scripts/engine.sh Raises default TRIAGE_TIMEOUT_SEC to reduce triage subprocess timeouts.
prompts/triage.md Strengthens prompt constraints to reduce non-JSON / conversational triage outputs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread prompts/triage.md
Comment on lines +14 to +15
with `"escalate": true` and a signal explaining what was missing — never
respond with prose.

Copilot AI Apr 30, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new fallback instruction says to emit JSON with "escalate": true and "a signal" explaining what's missing. Since the required schema below uses a signals array (and scripts parse .signals | join(...)), this wording risks the model outputting a signal field or omitting signals, which would break the contract. Suggest rephrasing to explicitly say to populate the signals array (and still include required risk and summary fields).

Suggested change
with `"escalate": true` and a signal explaining what was missing — never
respond with prose.
with `"escalate": true`, populate the `signals` array with what was missing,
and still include the required `risk` and `summary` fields — never respond
with prose.

Copilot uses AI. Check for mistakes.
don-petry added a commit that referenced this pull request Jun 7, 2026
Two recent failures in the PR Review Agent workflow each aborted the
hourly batch session-fatally:

- Run #461: triage model returned conversational text asking
  clarifying questions instead of JSON, breaking the contract.
- Run #470: triage process exited 1 on a single PR (likely timeout
  or subprocess issue with no diagnostic output).

Changes:
1. Bump TRIAGE_TIMEOUT_SEC default 180s → 300s. The 180s cap was
   tight for large PRs and offers no headroom for the model to
   complete; 300s matches ACTION_TIMEOUT_SEC and reduces the chance
   of a kill being misclassified as a deterministic failure.
2. Tighten the triage prompt: explicitly forbid clarifying questions,
   plan proposals, and prose. Add a fallback rule that says even on
   incomplete context the model must still emit JSON (with
   escalate=true) rather than respond conversationally.

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
don-petry added a commit that referenced this pull request Jun 7, 2026
Two recent failures in the PR Review Agent workflow each aborted the
hourly batch session-fatally:

- Run #461: triage model returned conversational text asking
  clarifying questions instead of JSON, breaking the contract.
- Run #470: triage process exited 1 on a single PR (likely timeout
  or subprocess issue with no diagnostic output).

Changes:
1. Bump TRIAGE_TIMEOUT_SEC default 180s → 300s. The 180s cap was
   tight for large PRs and offers no headroom for the model to
   complete; 300s matches ACTION_TIMEOUT_SEC and reduces the chance
   of a kill being misclassified as a deterministic failure.
2. Tighten the triage prompt: explicitly forbid clarifying questions,
   plan proposals, and prose. Add a fallback rule that says even on
   incomplete context the model must still emit JSON (with
   escalate=true) rather than respond conversationally.

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
don-petry added a commit that referenced this pull request Jun 8, 2026
Two recent failures in the PR Review Agent workflow each aborted the
hourly batch session-fatally:

- Run #461: triage model returned conversational text asking
  clarifying questions instead of JSON, breaking the contract.
- Run #470: triage process exited 1 on a single PR (likely timeout
  or subprocess issue with no diagnostic output).

Changes:
1. Bump TRIAGE_TIMEOUT_SEC default 180s → 300s. The 180s cap was
   tight for large PRs and offers no headroom for the model to
   complete; 300s matches ACTION_TIMEOUT_SEC and reduces the chance
   of a kill being misclassified as a deterministic failure.
2. Tighten the triage prompt: explicitly forbid clarifying questions,
   plan proposals, and prose. Add a fallback rule that says even on
   incomplete context the model must still emit JSON (with
   escalate=true) rather than respond conversationally.

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
don-petry added a commit that referenced this pull request Jun 12, 2026
Two recent failures in the PR Review Agent workflow each aborted the
hourly batch session-fatally:

- Run #461: triage model returned conversational text asking
  clarifying questions instead of JSON, breaking the contract.
- Run #470: triage process exited 1 on a single PR (likely timeout
  or subprocess issue with no diagnostic output).

Changes:
1. Bump TRIAGE_TIMEOUT_SEC default 180s → 300s. The 180s cap was
   tight for large PRs and offers no headroom for the model to
   complete; 300s matches ACTION_TIMEOUT_SEC and reduces the chance
   of a kill being misclassified as a deterministic failure.
2. Tighten the triage prompt: explicitly forbid clarifying questions,
   plan proposals, and prose. Add a fallback rule that says even on
   incomplete context the model must still emit JSON (with
   escalate=true) rather than respond conversationally.

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
don-petry added a commit that referenced this pull request Jun 14, 2026
Two recent failures in the PR Review Agent workflow each aborted the
hourly batch session-fatally:

- Run #461: triage model returned conversational text asking
  clarifying questions instead of JSON, breaking the contract.
- Run #470: triage process exited 1 on a single PR (likely timeout
  or subprocess issue with no diagnostic output).

Changes:
1. Bump TRIAGE_TIMEOUT_SEC default 180s → 300s. The 180s cap was
   tight for large PRs and offers no headroom for the model to
   complete; 300s matches ACTION_TIMEOUT_SEC and reduces the chance
   of a kill being misclassified as a deterministic failure.
2. Tighten the triage prompt: explicitly forbid clarifying questions,
   plan proposals, and prose. Add a fallback rule that says even on
   incomplete context the model must still emit JSON (with
   escalate=true) rather than respond conversationally.

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
don-petry added a commit that referenced this pull request Jun 18, 2026
Two recent failures in the PR Review Agent workflow each aborted the
hourly batch session-fatally:

- Run #461: triage model returned conversational text asking
  clarifying questions instead of JSON, breaking the contract.
- Run #470: triage process exited 1 on a single PR (likely timeout
  or subprocess issue with no diagnostic output).

Changes:
1. Bump TRIAGE_TIMEOUT_SEC default 180s → 300s. The 180s cap was
   tight for large PRs and offers no headroom for the model to
   complete; 300s matches ACTION_TIMEOUT_SEC and reduces the chance
   of a kill being misclassified as a deterministic failure.
2. Tighten the triage prompt: explicitly forbid clarifying questions,
   plan proposals, and prose. Add a fallback rule that says even on
   incomplete context the model must still emit JSON (with
   escalate=true) rather than respond conversationally.

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
don-petry added a commit that referenced this pull request Jun 25, 2026
Two recent failures in the PR Review Agent workflow each aborted the
hourly batch session-fatally:

- Run #461: triage model returned conversational text asking
  clarifying questions instead of JSON, breaking the contract.
- Run #470: triage process exited 1 on a single PR (likely timeout
  or subprocess issue with no diagnostic output).

Changes:
1. Bump TRIAGE_TIMEOUT_SEC default 180s → 300s. The 180s cap was
   tight for large PRs and offers no headroom for the model to
   complete; 300s matches ACTION_TIMEOUT_SEC and reduces the chance
   of a kill being misclassified as a deterministic failure.
2. Tighten the triage prompt: explicitly forbid clarifying questions,
   plan proposals, and prose. Add a fallback rule that says even on
   incomplete context the model must still emit JSON (with
   escalate=true) rather than respond conversationally.

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
don-petry added a commit that referenced this pull request Jun 25, 2026
Two recent failures in the PR Review Agent workflow each aborted the
hourly batch session-fatally:

- Run #461: triage model returned conversational text asking
  clarifying questions instead of JSON, breaking the contract.
- Run #470: triage process exited 1 on a single PR (likely timeout
  or subprocess issue with no diagnostic output).

Changes:
1. Bump TRIAGE_TIMEOUT_SEC default 180s → 300s. The 180s cap was
   tight for large PRs and offers no headroom for the model to
   complete; 300s matches ACTION_TIMEOUT_SEC and reduces the chance
   of a kill being misclassified as a deterministic failure.
2. Tighten the triage prompt: explicitly forbid clarifying questions,
   plan proposals, and prose. Add a fallback rule that says even on
   incomplete context the model must still emit JSON (with
   escalate=true) rather than respond conversationally.

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants