Let the reviewer pin findings and inline comments to diff lines - #596
Open
JoviDeCroock wants to merge 14 commits into
Open
Let the reviewer pin findings and inline comments to diff lines#596JoviDeCroock wants to merge 14 commits into
JoviDeCroock wants to merge 14 commits into
Conversation
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
staged-publish-review | c670003 | Commit Preview URL Branch Preview URL |
Aug 22 2026, 05:30 AM |
Detection eval report
Regression (gated)
Frontier (reported — truth-labeled hard cases)recall 91% (10/11)
Benign hard-negatives (reported — false-positive precision)false positives 0/10 (0%) Evasion robustness (reported)
Updated by CI for this PR. |
JoviDeCroock
force-pushed
the
JoviDeCroock/karachi-v2
branch
2 times, most recently
from
August 21, 2026 12:32
96c2ae0 to
b1407f2
Compare
The AI reviewer could only report whole-file findings, so every assistant signal landed in the diff's unpinned banner instead of on the hunk that triggered it, and every observation below critical/high vanished into the summary prose. The reviewer now submits an `anchor` — a line copied verbatim from the evidence it was served — and `resolveAnchorLine` matches that string against the same text sample, requiring a unique match. It never states a line number: the evidence tools serve unnumbered text, so a number would be counted from memory, and a mis-pinned finding points a maintainer at innocent code while quietly exonerating the guilty line. An anchor that misses, matches several lines, or is too generic resolves to no line and falls back to the banner. Findings carry the resolved line through both stores into `scan_findings.line`, so an assistant finding pins to its hunk like a deterministic one. A review may also return up to six `comments`: severity-free notes pinned to a line, persisted in `scans.ai_json`, rendered with a neutral badge and the info tone because a note is not a signal. They are never `scan_findings` rows, never count into `finding_count`, and cannot move risk. A resolved line is display only. `annotateFindingsWithDiffStatus` ignores it for `source: "ai"` findings — otherwise a whole-file concern anchored to an untouched line would fall out of the release bucket `releaseRisk` and the workflow gate read, letting a display feature lower the gate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`AI_REVIEWER_VERSION` identifies the prompt, evidence tools, model routing, and response contract as one unit, and its own comment requires a bump whenever a change can alter reviewer behavior. This branch rewrites the system prompt with two new sections, adds `anchor` and `comments` to the submission schema, and raises MAX_REVIEW_OUTPUT_TOKENS — all behavior-altering — while leaving the stamp at 1.2.0. Persisting the old version is exactly the failure the constant exists to prevent: analytics and recorded eval cases would compare two different reviewer contracts as though they were one. Re-stamps the recorded eval corpus onto 1.3.0 in the same change, which is the procedure docs/ai-review-eval.md prescribes — the harness gates on the current version, so the bump and the records have to move together. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
JoviDeCroock
force-pushed
the
JoviDeCroock/karachi-v2
branch
from
August 22, 2026 05:30
b1407f2 to
c670003
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The AI reviewer can now pin its findings — and leave short advisory comments — on specific lines of the diff, instead of every assistant note floating above the hunks in a banner.
The mechanism is deliberately narrow: the model never states a line number. It submits an
anchor— one line copied verbatim out of the evidence it was served — andserver/lib/ai-review/anchors.tsresolves that string against the same text sample, requiring a unique match. An anchor that misses, matches several lines, or is too generic to identify one (},});) resolves to no line and falls back to the existing unpinned banner. Matching a string the model must have actually seen is the only claim we can check, so it's the only claim we accept — a mis-pinned finding is worse than an unpinned one, because it points a maintainer at innocent code and quietly exonerates the guilty line.Two kinds of annotation ride that machinery:
anchor, resolved to alinethat is persisted on thescan_findingsrow and captionedassistant · line N.assistant notein the neutral info tone. They live inscans.ai_jsononly — neverscan_findingsrows, never infinding_count, never in Risk signals or the file-tree counts.Why a pinned line can't move the verdict
A resolved line is display-only.
annotateFindingsWithDiffStatusnow scopessource: "ai"findings by file rather than by line, because the reviewer pins a concern to the clearest line illustrating it — which for a whole-file or manifest-wide argument is often a line the release never touched. Reading that coordinate as line-level evidence would move an AI finding out of the release bucket thatreleaseRiskand the workflow gate score: a display feature quietly lowering risk. AI findings stay scoped exactly as they were before anchors existed.Comments cannot move risk at all. A comment naming a file the review could not see is dropped outright; a finding in the same position is kept, because its evidence still stands on its own.
Contract changes
anchor(optional, per finding) andcomments(optional, capped) added to the submission schema, with matching prompt sections andAI_REVIEW_BOUNDSentries.MAX_REVIEW_OUTPUT_TOKENS8k → 10k, so a worst-case submission still serializes without truncating mid-JSON.AI_REVIEWER_VERSION1.2.0 → 1.3.0, with the recorded eval corpus re-stamped in the same change perdocs/ai-review-eval.md.lineandcommentsas optional/defaulted, so every review written before this branch still parses.Testing
pnpm run verifygreen (lint, format, typecheck, tests).test/ai-review-anchors.test.mjs(anchor resolution — exact/partial/marker-stripped/ambiguous/too-generic), anchor + comment coverage intest/ai-review.test.mjs, file-scoping of AI findings intest/review.test.mjs, caption behavior intest/diff-annotations.test.ts, and persistence/export coverage intest/workers/.docs/release-safety.md(anchor contract, comments-are-not-signals) anddocs/ui.md(the three annotation kinds).The final commit is a self-review fix: the branch changed the prompt, the schema, and the output budget while leaving
AI_REVIEWER_VERSIONat 1.2.0, which is exactly the drift that constant exists to prevent.🤖 Generated with Claude Code