Implement Claude generic code review for GitHub

[don-petry]

To add more voices in our LLM as judge PR review process, add the Claude standard code review automation to our org. This adds more signals and distributes the load across more providers.

Scope: all repos
Org Standard: yes
Replaces? No, additive
Token: Uses current org secret for Claude

https://code.claude.com/docs/en/code-review

[github-actions[bot]]

Enhancement: Claude Code Review — Multi-Reviewer Architecture Analysis

Sharper Problem Framing

The core value proposition is diversifying the AI review signal — multiple independent models catch different classes of bugs. The org already runs pr-auto-review (Claude-based, dispatched via ready-check) and has evaluated Qodo Merge (#705). Adding Claude Code Review as a managed GitHub App introduces a second structured review signal. The question is whether the incremental signal justifies the cost and cognitive load.

Prior Art & Competitive Landscape

Claude Code Review is a managed, multi-agent review service in research preview (Team + Enterprise plans):

Dimension	Claude Code Review (App)	Existing pr-auto-review	CodeRabbit	GitHub Copilot Review
Delivery	GitHub App (managed)	Claude Code CLI via Actions	GitHub App	Native GitHub feature
Trigger	PR open + every push + @claude review	Ready-check gate (all CI green, no blocking reviews)	PR open + push	PR open + @copilot review
Architecture	Multi-agent fleet (parallel specialized analyzers + verification step)	Single agent with full codebase context	Single agent	Single agent
Findings format	Severity-tagged inline comments (Important / Nit / Pre-existing)	Structured review comment (pass/fail + detailed findings)	Inline comments + summary	Inline suggestions
Configuration	REVIEW.md (review-only instructions) + CLAUDE.md (project context)	Agent prompt in workflow	.coderabbit.yml	Copilot settings
Cost	~$15–25/PR (token-based, usage-billed)	Anthropic API tokens (controlled via org secret)	~$24/dev/month (flat)	Included with Copilot subscription
Never does	Approve or block PRs	Can request changes (configurable)	Can approve (configurable)	Suggests only

Key differentiator: Claude Code Review runs a verification step that filters false positives before posting — designed to reduce the "AI review noise" problem that plagues multi-reviewer setups.

Options Table

Option	Architecture	Pros	Cons	Est. Monthly Cost
A. Add alongside pr-auto-review	Both run on every PR; partition by responsibility	Maximum coverage; different agent architectures catch different bugs	Risk of duplicate/conflicting findings; $15-25/PR adds up; cognitive load on reviewers	$300-750 (est. 20-30 PRs/month)
B. Replace pr-auto-review	Claude Code Review App as sole AI reviewer	Simpler stack; managed service reduces maintenance; multi-agent architecture may exceed single-agent depth	Loses the ready-check gate (Code Review triggers on PR open, not CI-green); lose custom prompt control	$300-750
C. Claude Code Review as secondary, scoped	pr-auto-review remains primary; Code Review App enabled for security-critical paths only (via REVIEW.md path filters)	Targeted coverage; lower cost; additive signal where it matters most	Partial coverage; configuration overhead to avoid overlap	$100-250
D. Wait for GA	Defer until research preview graduates	Avoid preview-quality risks; pricing may change; feature set will mature	Miss early adoption learning; review quality may already be production-grade	$0

Risks & Unknowns

1. Research preview status. No SLA, features may change, pricing may increase at GA. The org would be building standards around a preview product.
2. Review fatigue is real. Industry research shows developers tune out when AI reviewers produce >5 comments per PR. Running two AI reviewers without careful scoping amplifies this.
3. Cost at scale is unpredictable. Token-based pricing means large PRs or PRs with many pushes cost significantly more. The "after every push" trigger can multiply costs by 3-5x per PR.
4. Overlap with pr-auto-review. Both use Claude models. Without careful REVIEW.md scoping, they will flag the same issues in different formats.
5. The ready-check gate is valuable. pr-auto-review only fires when CI is green and no blocking reviews exist. Claude Code Review fires immediately on PR open — reviewing code that may still be failing tests.

Multi-Reviewer Best Practices (If Adopting)

• Partition responsibilities: Use REVIEW.md to scope Code Review to security, edge cases, and logic bugs. Configure pr-auto-review to focus on standards compliance, architecture patterns, and test coverage.
• Cap nit volume: REVIEW.md supports limiting low-severity comments. Set a ceiling (e.g., max 3 nits per review).
• Skip CI-covered checks: Exclude linting, formatting, and type errors from both reviewers — CI already catches these.
• Monitor developer reaction: Both tools support feedback (thumbs up/down). Track acceptance rate monthly; if <50%, reduce scope.

Phased Validation Path

1. Week 1: Install the Claude GitHub App on a single test repo. Create a REVIEW.md scoping reviews to security and logic bugs only. Run 5-10 PRs and measure: finding overlap with pr-auto-review, false positive rate, developer feedback.
2. Week 2: Compare cost per PR against the current pr-auto-review token spend. Model org-wide cost.
3. Week 3: Decision gate: adopt (Option A or C), replace (Option B), or defer (Option D).
4. If adopting: Document the multi-reviewer standard in standards/ci-standards.md per the org's reusable-workflow model. Define which repos get which reviewers.

Success Metrics

• Trial gate: False positive rate ≤15%; ≥30% of findings are unique (not caught by pr-auto-review).
• Cost gate: Per-PR cost ≤$20 average for the org's typical PR size.
• Adoption gate: Developer acceptance rate (thumbs-up on findings) ≥60% after 30 days.