Skip to content

feat: anomaly detector — flag outlier or suspicious pending proposals for closer review #323

Description

@plind-junior

reviewers scan the pending queue linearly, and at a glance every item carries the same apparent weight. the ones that most deserve a hard look — a claim far from anything already in the kb, a claim that contradicts a pile of approved ones, a claim scraping by with the barest evidence — are exactly the ones that blend in. the raw signal for the near-duplicate direction already exists (find_similar_on_propose attaches non-blocking warnings on propose_claim, and embeddings/dedup.py runs cross-artifact cosine scans over embedding_index), but nothing surfaces the outlier direction: proposals that are anomalous rather than redundant.

this asks for an advisory anomaly annotation on pending proposals. it computes a small set of heuristics — nearest-neighbor cosine distance to the approved corpus (an item with no neighbor above a floor is an outlier), a contradiction count against approved claims, and an evidence-thinness signal — and attaches the result as a read-side annotation on the pending item. it is a hint for the human reviewer. it never rejects, approves, blocks, or rewrites anything.

proposed surface

a read-only kb.flag_anomalies method plus a --flag-anomalies opt-in on the pending listing:

  • vouch flag-anomalies [--threshold N] [--json] — scores every pending proposal and prints the flagged ones with their reason codes, worst-first. read-only; no writes.
  • vouch list-pending --flag-anomalies — the same scores attached inline to each pending row, so a reviewer sees them without a second command.
  • the annotation reuses the warning-dict shape already returned by find_similar_on_propose — a list of {code, ...} entries per proposal. proposed codes:
    • far_from_corpus — best cosine to any approved artifact is below a floor (nearest-neighbor is far). computed against the same search_embedding / embedding_index path embeddings/dedup.py uses.
    • contradicts_many — count of approved claims this proposal would contradict (over kb.contradict's existing notion) exceeds a threshold.
    • thin_evidence — evidence list is at or below a floor. propose_claim already requires at least one evidence id; this is the softer "technically cited but suspiciously thin" case (see bug: Claim model has no min-evidence validator — uncited claims land via bundle import, put_claim, update_claim #81, below).
  • config lives under the existing review.* block in .vouch/config.yaml (mirroring review.similarity_threshold): e.g. review.anomaly.far_from_corpus_floor, review.anomaly.contradiction_count, review.anomaly.min_evidence. resolution follows the similarity_threshold(store) pattern in embeddings/similarity.py.
  • embedding-derived codes degrade gracefully when the embeddings extra is absent — the same ImportError swallow propose_claim already does around find_similar_on_propose, so a base install still gets the non-embedding codes (thin_evidence, contradicts_many).

review gate & scope

this is a read-side advisory only. it computes nothing durable: no proposal, no artifact, no status transition. the flag is presentation metadata attached at read time, exactly like the warnings on propose_claim and unlike anything that writes. because it emits no proposal and touches no artifact, it goes nowhere near proposals.approve() — and it must never gain a "quarantine" or "auto-reject outliers" mode, which would be a write path bypassing the human gate. background or scheduled callers may run it to compute hints; they still cannot act on them, and no code path here approves or rejects. everything stays local: the scores come from the on-disk kb and the local state.db embedding index, no network.

scoring logic (thresholds, contradiction counting, corpus-distance) belongs in a new scoring helper called from the read handlers, not in storage.py, which stays pure I/O.

adding kb.flag_anomalies touches the four registration sites: @mcp.tool() in src/vouch/server.py, _h_flag_anomalies + HANDLERS["kb.flag_anomalies"] in src/vouch/jsonl_server.py, METHODS in src/vouch/capabilities.py, and the vouch flag-anomalies command in src/vouch/cli.py, plus tests/test_flag_anomalies.py. (if the --flag-anomalies flag on kb.list_pending alone covers the need, only the flag + tests are required; decide during design.)

adjacent issues, with the distinction stated:

acceptance criteria

  • pending proposals can be scored for anomaly and returned with per-proposal reason codes, worst-first, read-only.
  • far_from_corpus reuses the existing search_embedding / embedding_index path and produces no code when the embeddings extra is absent.
  • thin_evidence and contradicts_many are computed without embeddings so a base install still gets non-embedding flags.
  • thresholds resolve from review.anomaly.* in .vouch/config.yaml, defaulting sanely when unset, following the similarity_threshold(store) resolution pattern.
  • the feature emits no proposal, writes no artifact, and performs no status transition — verified by an audit-log assertion that a scoring run adds no mutation events.
  • scoring logic lives outside storage.py (a scoring helper), keeping the I/O layer pure.
  • if a new kb.* method is added, all four registration sites are updated with parity and test_capabilities passes.
  • tests/test_flag_anomalies.py covers: an obvious outlier flagged, a normal in-cluster claim not flagged, thin-evidence flagged, graceful degradation with no embedder, and the no-mutation invariant.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestsize: M200-499 changed non-doc lines

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions