You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Confidence already drives the review verdict but not comment placement. ReviewState.java:35-41 escalates to REQUEST_CHANGES only for (CRITICAL || HIGH) && confidence == HIGH; everything else is a COMMENT. But every finding — at any confidence — is still posted as an inline review thread. SuggestionFormatter.java:68-71 merely appends a "(<level> confidence — verify before acting)_" disclaimer; it does not change where, or whether, the finding is posted.
So a low-confidence finding still opens an inline thread that reads as actionable and that a maintainer must triage and resolve. The deterministic guards (FindingQuoteValidator.withoutSuggestion at :204-214) and any future prompt self-checks can only cap confidence to low — and that capping barely helps today, because a low-confidence finding lands inline anyway.
Dogfood evidence — PR #101: the NPE finding on DashboardAccessChecker.java:222 was emitted at low confidence yet posted as a MEDIUM inline thread; a maintainer had to read it, refute it, and resolve the thread (PR #101discussion_r3410353985). Net cost: noise plus manual resolution for a finding the bot itself was not sure about.
Proposed Solution
Gate inline placement on confidence, reusing the existing ordered Confidence enum:
Post an inline thread only for findings with confidence >= MEDIUM (keep risk >= HIGH inline too, so genuinely severe-but-uncertain items stay visible).
Route low-confidence findings to a collapsed "Things to double-check" section of the PR summary (PrSummaryGenerator), with file/line and the existing disclaimer — visible and clearly non-blocking, but not an inline thread that demands resolution.
Leave ReviewState verdict logic unchanged (it already requires confidence == HIGH to block).
Cover with a unit test asserting low-confidence findings are excluded from the inline set and appear in the summary section.
This makes "cap confidence to low" a meaningful lever: the deterministic validators and prompt self-checks that downgrade shaky findings then actually move them out of the inline stream.
Alternatives Considered
Suppress low-confidence findings entirely — loses occasionally-useful signal and hides what the bot considered.
Problem Statement
Confidence already drives the review verdict but not comment placement.
ReviewState.java:35-41escalates toREQUEST_CHANGESonly for(CRITICAL || HIGH) && confidence == HIGH; everything else is aCOMMENT. But every finding — at any confidence — is still posted as an inline review thread.SuggestionFormatter.java:68-71merely appends a"(<level> confidence — verify before acting)_"disclaimer; it does not change where, or whether, the finding is posted.So a low-confidence finding still opens an inline thread that reads as actionable and that a maintainer must triage and resolve. The deterministic guards (
FindingQuoteValidator.withoutSuggestionat:204-214) and any future prompt self-checks can only cap confidence to low — and that capping barely helps today, because a low-confidence finding lands inline anyway.Dogfood evidence — PR #101: the NPE finding on
DashboardAccessChecker.java:222was emitted at low confidence yet posted as a MEDIUM inline thread; a maintainer had to read it, refute it, and resolve the thread (PR #101discussion_r3410353985). Net cost: noise plus manual resolution for a finding the bot itself was not sure about.Proposed Solution
Gate inline placement on confidence, reusing the existing ordered
Confidenceenum:confidence >= MEDIUM(keeprisk >= HIGHinline too, so genuinely severe-but-uncertain items stay visible).PrSummaryGenerator), with file/line and the existing disclaimer — visible and clearly non-blocking, but not an inline thread that demands resolution.ReviewStateverdict logic unchanged (it already requiresconfidence == HIGHto block).This makes "cap confidence to low" a meaningful lever: the deterministic validators and prompt self-checks that downgrade shaky findings then actually move them out of the inline stream.
Alternatives Considered
Priority
Nice to have
Additional Context
SuggestionFormatter.java:68-71(disclaimer, posted regardless of confidence);ReviewState.java:35-41(verdict already confidence-aware);Confidence.java(ordered enum);PrSummaryGenerator(summary target).Implementation order
Code of Conduct