[LOW] feat(review): lint findings whose confidence outruns their evidence

[elhoim]

BLUF

• Problem — every finding carries confidence (HIGH/MEDIUM/LOW) and confidence_justification, both written by the same generative model that wrote the finding. A model's self-report of its own certainty is not calibrated, and the examiner is shown it as though it were.
• Change — vhir review --findings --lint flags findings whose stated confidence outruns their cited evidence. Seven checks, all over surface features of the finding.
• Scope, stated plainly — this is a lint, not a calibration. It does not judge whether the evidence supports the claim; that needs reading the cited artifacts. A flag is a prompt to look, and an unflagged finding has not been verified.
• Never overwrites — the finding's own confidence is shown beside the flags, unchanged. Nothing reads or writes status.
• Lowest-value of the four proposals, and honestly labelled as such — see "Where this degrades" below.

Priority: LOW

Needs a decision before merge. This is the one proposal of four that does not survive the removal of a semantic judgment intact. It is worth having, but it is worth having under an accurate name. The decision is written up in #63.

Measured

Three findings, all stating HIGH confidence:

2 of 3 finding(s) flagged.

  [F-002] Log clearing  (stated confidence: HIGH)
      confidence-without-evidence: stated HIGH confidence but cites no audit_ids
      unsupported-intent: interpretation asserts intent ('in order to') with nothing cited
      attribution-claim: interpretation makes an attribution claim ('ransomware group')
      interpretation-exceeds-observation: interpretation is 7.3x the observation with nothing cited
      unjustified-confidence: stated HIGH confidence with no confidence_justification
      hedged-but-high: stated HIGH confidence but the interpretation hedges ('consistent with')

  [F-003] Odd port  (stated confidence: HIGH)
      confidence-without-evidence: stated HIGH confidence but cites no audit_ids
      hedged-but-high: stated HIGH confidence but the interpretation hedges ('likely')

Surface checks only — a flag is a prompt to look, not a finding of error,
and an unflagged finding has not been verified. Stated confidence is unchanged.

F-001 — same HIGH confidence, but it cites audit_ids, its justification is filled in, and its interpretation is one unhedged sentence — is not flagged.

F-002 is the case worth having: "the log was cleared in order to evade detection by the incident response team, consistent with a known ransomware group affiliate playbook", from the observation "Security log cleared.", at HIGH confidence, citing nothing. Every one of those is checkable without understanding the incident.

The checks

Code	Raised when
confidence-without-evidence	HIGH confidence, audit_ids empty
unjustified-confidence	HIGH confidence, no confidence_justification
unsupported-intent	Interpretation asserts purpose (in order to, intended to, to evade…) with nothing cited
attribution-claim	Interpretation names an actor class (ransomware group, APT, nation-state) — always flagged, cited or not
interpretation-exceeds-observation	Interpretation is >2.5× the observation by word count, with nothing cited
hedged-but-high	HIGH confidence, but the interpretation hedges (likely, appears to, consistent with)
no-observation	Observation empty

attribution-claim fires regardless of citations on purpose: attribution is a claim a single host's artifacts essentially never support, so it deserves a human look every time.

Where this degrades

The original proposal asked a model to score evidence_supports_interpretation — judging the interpretation against its own observation and cited audit_ids, with levels from "the observation alone establishes this" to "the interpretation asserts intent the observation cannot show".

That is not what this does, and I do not want the name to imply otherwise. Deterministically:

• Whether a cited audit_id actually contains what the interpretation claims — not checkable here. It needs reading the artifact.
• Whether an interpretation is a reasonable reading of its observation — not checkable here. It needs understanding both.
• Whether something was cited at all, whether the language claims more than an observation can carry, whether the confidence field contradicts the prose — checkable, and that is the whole of what this ships.

So: renamed from "calibrated confidence" to evidence-support lint. It is the honest version of the proposal, and it is the smallest of the four in value. It goes in the same box as a linter that flags a bare except: — it does not know your code is wrong, it knows the shape is worth a second look.

Two guards against over-reading it are built into the output: the trailing note says an unflagged finding has not been verified, and stated confidence is printed next to every flag so the examiner sees what the model claimed.

Optional use in ranking

If #57 (review-queue ranking) lands, a flagged finding is a natural candidate to sort up — the original proposal's point that disagreement between the writer's confidence and an independent check is the useful signal. Not wired here; the two PRs stay independent.

Testing

tests/test_evidence_lint.py, 18 tests: each check firing and, more importantly, each check not firing when it should not (intent language with citations, hedging at MEDIUM, verbosity with citations), a well-supported finding raising nothing, findings never mutated, status and confidence untouched, deterministic flag order, and whole-word matching of the attribution terms (apt must not fire on "captured").

• pytest tests/ — 446 passed on Python 3.10 and 3.12 (428 pre-existing, unchanged; 18 new)
• ruff check . / ruff format --check . — clean

Deliberately not in this PR

• Any change to confidence. Overwriting a model's self-report with a different automated number would replace one uncalibrated signal with another and lose the disagreement, which is the informative part.
• Blocking or filtering on flags. Nothing here gates approval. A finding that trips every check is still shown, still approvable, still the examiner's call.
• Tuning the word lists. They are plain tuples at the top of the module. They encode English, Windows-centric IR prose, which is what this tool's findings are written in.

---

CI status

The checks will show action_required: GitHub gates workflow runs on PRs from first-time fork contributors until a maintainer approves them. Reproduced locally with the same steps as ci.yml:

Step	Result
pytest tests/ — Python 3.10	446 passed
pytest tests/ — Python 3.12	446 passed
ruff check .	All checks passed
ruff format --check .	clean

Branched from main @ 954a623.

Cherry-pick note. Verified against every other PR in this series, not assumed.

• Onto [HIGH] fix(iocs): validate domain IOCs against the IANA TLD list #54, [HIGH] fix(iocs): extract defanged indicators from finding text #55 or [MEDIUM] feat(iocs): classify extracted addresses by routability #58: import line in review.py only. Keep both lines.
• Onto [MEDIUM] feat(review): group near-duplicate findings under one review unit #56 or [MEDIUM] feat(review): rank the review queue by investigative value #57: the import, the p_review flag block in main.py, the dispatch branch, and the point where each PR inserts its new view function into review.py. All "keep both"; the only judgement is dispatch precedence.
• See the full-series check below.

Full-series check. All six PRs in this series (#54, #55, #56, #57, #58, #59) were combined into one tree and gated together, not assumed compatible: 573 passed on Python 3.10 and 3.12, ruff clean, and all three new review --findings views reachable through the merged dispatch. Every conflict between them is additive ("keep both"), concentrated in the import block, the p_review flag list and the view dispatch.

🤖 Generated with Claude Code

[scriptedstatement]

Thanks. We're deferring new review-queue features to the next minor release. No problems found in review; it's a product decision about the review queue that we'll make then.