Skip to content

[LOW] Decision: does an evidence-support lint belong in the CLI, or upstream at staging? (PR #59) #63

Closed as not planned

Description

@elhoim

BLUF — what needs deciding

Does an evidence-support lint belong in the CLI at all, or upstream at staging time?

  • PR [LOW] feat(review): lint findings whose confidence outruns their evidence #59 adds vhir review --findings --lint, flagging findings whose stated confidence outruns their cited evidence: HIGH confidence with no audit_ids, an interpretation asserting intent or attribution with nothing cited, an interpretation far longer than its observation, HIGH confidence alongside hedging.
  • The underlying problem is real and is not in dispute: confidence and confidence_justification are written by the same generative model that wrote the finding, and the examiner is shown a self-report as though it were calibrated.
  • The honest limitation is the decision. This is a lint over surface features of the text. It does not and cannot judge whether the cited evidence supports the claim — that needs reading the artifacts. A flagged finding may be perfectly sound; an unflagged one has not been verified. Both caveats are printed with the output, but a reader can still over-trust it.

Options

  1. Ship as-is, in review. Cheap, opt-in, honest about its limits. Risk: an examiner reads "nothing flagged" as "checked".
  2. Move it upstream to staging. The MCP backend writes these findings; flagging there means the model gets told its confidence is unsupported before a human ever sees the finding. Arguably the right place — the CLI is then reviewing better input rather than annotating bad input.
  3. Don't ship it. If you judge that a surface lint invites more false confidence than it removes, that is a defensible call and I would not argue hard against it. This is the weakest of the four proposals in this series and I would rather say so than oversell it.

If you ship it

  • Should flags feed [MEDIUM] feat(review): rank the review queue by investigative value #57's ranking, sorting flagged findings up? The disagreement between a writer's stated confidence and an independent check is the informative signal. Not wired; the two PRs are independent.
  • The word lists (_INTENT_TERMS, _ATTRIBUTION_TERMS, _HEDGE_TERMS) encode English, Windows-centric IR prose. They are plain tuples at the top of the module.
  • attribution-claim fires whether or not evidence is cited, on the grounds that a single host's artifacts essentially never support attribution. Tell me if that is too aggressive.

Related: #56, #57, #58 — same design review, each independently cherry-pickable.

Activity

  1. scriptedstatement commented on Oct 13, 2026

    @scriptedstatement
    Contributor

    Closed with #59; see the comment there. Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions