Skip to content

feat(review): gate inline posting on confidence — route low-confidence findings to the summary #105

Description

@devops-thiago

Problem Statement

Confidence already drives the review verdict but not comment placement. ReviewState.java:35-41 escalates to REQUEST_CHANGES only for (CRITICAL || HIGH) && confidence == HIGH; everything else is a COMMENT. But every finding — at any confidence — is still posted as an inline review thread. SuggestionFormatter.java:68-71 merely appends a "(<level> confidence — verify before acting)_" disclaimer; it does not change where, or whether, the finding is posted.

So a low-confidence finding still opens an inline thread that reads as actionable and that a maintainer must triage and resolve. The deterministic guards (FindingQuoteValidator.withoutSuggestion at :204-214) and any future prompt self-checks can only cap confidence to low — and that capping barely helps today, because a low-confidence finding lands inline anyway.

Dogfood evidence — PR #101: the NPE finding on DashboardAccessChecker.java:222 was emitted at low confidence yet posted as a MEDIUM inline thread; a maintainer had to read it, refute it, and resolve the thread (PR #101 discussion_r3410353985). Net cost: noise plus manual resolution for a finding the bot itself was not sure about.

Proposed Solution

Gate inline placement on confidence, reusing the existing ordered Confidence enum:

  • Post an inline thread only for findings with confidence >= MEDIUM (keep risk >= HIGH inline too, so genuinely severe-but-uncertain items stay visible).
  • Route low-confidence findings to a collapsed "Things to double-check" section of the PR summary (PrSummaryGenerator), with file/line and the existing disclaimer — visible and clearly non-blocking, but not an inline thread that demands resolution.
  • Leave ReviewState verdict logic unchanged (it already requires confidence == HIGH to block).
  • Cover with a unit test asserting low-confidence findings are excluded from the inline set and appear in the summary section.

This makes "cap confidence to low" a meaningful lever: the deterministic validators and prompt self-checks that downgrade shaky findings then actually move them out of the inline stream.

Alternatives Considered

Priority

Nice to have

Additional Context

Implementation order

  1. feat(review): gate inline posting on confidence — route low-confidence findings to the summary #105 (this) — gate inline posting on confidence. Do first: it makes "cap to low" actually reduce noise, which is the lever fix(review): FindingQuoteValidator checks only suggestion_old, so fabricated code in the description passes #106 and feat(review): cap/reject precondition (null-deref) findings whose parameter source isn't in the diff #107 pull.
  2. fix(review): FindingQuoteValidator checks only suggestion_old, so fabricated code in the description passes #106 (deterministic, stronger) and feat(review): cap/reject precondition (null-deref) findings whose parameter source isn't in the diff #107 (prompt rule) — independent near-term guards; either order.
  3. feat(review): codebase-aware context beyond the diff (cross-file analysis) #55 — codebase-aware context: the structural fix that makes precondition/cross-file claims verifiable; subsumes feat(review): cap/reject precondition (null-deref) findings whose parameter source isn't in the diff #107 and enriches fix(review): FindingQuoteValidator checks only suggestion_old, so fabricated code in the description passes #106. Larger effort / its own spike.

Code of Conduct

  • I have searched for existing issues and this is not a duplicate.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestjavaPull requests that update java code

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions