Repository navigation
Confidence chooses the channel: uncorroborated findings stop paging and land in the digest — unranked alerts cost the attention the true positives need #3712
Description
Confidence measures corroboration now — that was the point of the scorer rework — but every finding that crosses the severity floor still takes the same road to the same channel. The result, measured on the first day the recalibrated engine ran in production: forty-plus anomaly pages in seven hours, severity 1.5–1.9, confidence 0.20–0.35, interleaved with the handful of pages that actually needed a human. The operator's own reading of that channel: no idea what to focus on. That is the alert-fatigue mechanism in one sentence — not wrong alerts, but unranked ones, where the cost is paid by the true positives that stop being seen.
The batch itself makes the case sharper, because the low-confidence pages were largely true and redundant: anomaly stories attaching to a CPU flap that was already paging through the CPU family, to a server in the middle of its own maintenance job, to an ACU burst the platform CPU alert had already delivered. The anomaly engine found real things — things something else had already announced. Uncorroborated single-fact stories are, empirically, mostly echoes.
Proposal: confidence chooses the channel
- Page (webhook/Teams/PagerDuty): findings with real corroboration — two or more facts in the chain, or a genuine co-fire. The extremity escape keeps its severity power, but its published intent (extreme and corroborated) becomes the enforced bar.
- Digest + web (never a page): lone-fact / low-corroboration findings. They appear in the daily digest as a counted, grouped section ("anomaly singles: N across M servers, top movers…") and remain fully visible on the web surface and MCP reads the moment they fire. Nothing is suppressed; nothing interrupts.
- The alert-history ledger records the routing decision the same way it records severity_source, so "why didn't this page" is answerable from the row.
Boundaries with in-flight work
Two sibling efforts reduce the volume of this class and are deliberately out of scope here: the dented-baseline follow-up (restart zeros inflating z-scores — decays naturally and has its own lane) and the RUNNING_JOBS fold's missing anomaly-path arm (redundant stories during a known job). This issue is the third leg and is needed even after both land: at steady state across a large fleet, single-fact z-score excursions will fire routinely by construction, and routing — not thresholds — is what keeps them from costing attention they haven't earned.
Design points to settle in review
- The gate reads the corroboration components of confidence, not the raw scalar, so a future re-weighting doesn't silently move the paging bar.
- Resolution behavior: a digest-routed finding that later gains corroboration (fact joins the chain, co-fire arrives) pages at that moment as a new firing — escalation is the corroboration event.
- The digest section carries the same dedup keys as pages would have, so an operator can promote any single to a full drill-down without archaeology.
Activity
Three pins the Mac could not run: persist the Lite knob as root[...] …
Confidence chooses the channel: uncorroborated findings route to dige…
erikdarlingdata commentedon Sep 19, 2026
Landed in #3732 (0998e6d) — items 1–6; the issue stays open for one residual.
Shipped:
FindingRouting.Classifyreads the corroboration COMPONENTS (fact count ≥ 2, a matched co-fire amplifier on the root, or one of the two by-construction stories) — never the scalar, so re-weighting cannot move the paging bar (design point 1). Digest-routed findings persist and remain on every read surface; the ledger row carriesnotification_type: digestwithrouting/routing_reason(design point 3's dedup fingerprints ride in the Analysis Singles Digest's top movers). Escalation is the corroboration event: the digest road keys its cooldown bucket under its own namespace and the page seed excludes digest rows in both stores, so a single that gains a fact pages as a new firing (design point 2; pinned red-then-green). One knob,analysis.uncorroborated_route, defaultsdigest.Residual (why Partial): the Darling knob is FILE-LEVEL only (
analysis.uncorroboratedRoutein darling.json, hot-reloaded). The store-backed setting + Viewer Settings surface needs aconfig_alert_settingscolumn — a rung, fenced while two rungs were in flight. Follow-up lane when the ladder is quiet: V135 candidate, one column, Viewer Settings → Alerts toggle,get_alert_settings.analysis.uncorroborated_routenon-null on the store path.One clarification from the lane: the extremity escape (
FactScorer.IsExtremeAnomaly) writes no marker on the fact, so the gate treats an extremity-escaped SINGLE as a single — it routes to the digest. That is the issue's stated intent ("extreme and corroborated") enforced at the routing layer; if an extreme single should page on its own, that is a scorer change (marker + gate arm), not this lane's.Acceptance on the next install: the monitor seat's burst curve, read per leg — pages vs digest singles is this issue's.
CHANGELOG: second splice - sixty-three entries from sixty-five merged…
Darling V137 / Lite v64: query_store_health stores each database's Qu…
The uncorroborated-finding route knob gets its store home's code half (…
Lite's McpAlertTools comments described Darling's provenance key by i…
A "read by nothing" pin cannot outlive the readers it was waiting for…
A "read by nothing" pin cannot outlive the readers it was waiting for…
The uncorroborated-finding route knob said FILE-LEVEL, edit darling.j…
erikdarlingdata commentedon Sep 21, 2026
Closed by the watcher: delivered in PR #3828, merged to dev.
erikdarlingdata commentedon Sep 22, 2026
Instrument 8 (Analysis Singles Digest never observed, 54 h) — CLOSED as nothing-to-digest, read from all three stores 2026-09-22 06:4xZ: zero
notification_type = 'digest'ledger rows since the build carrying #3732 installed. Every single that reached the notify floor in that span had ≥ 1 matched co-fire check and paged as corroborated (a 0.44-confidence single is exactly one match of two —0.20 + 0.48 × ½); the digest posts nothing on an empty day by design. Wiring verified end to end at source (routing →RoutedToDigest→ ledgernotification_type→AnalysisSinglesDigestReader→ the worker's hourly self-alert block, stamp-gated daily). One fold-in rider queued for the next lane in the evaluator: an unconditional daily INFORMATION line (N digest-routed rows — posted | nothing to post) so silence is answerable from the log. The store-backed knob residual landed in #3828; this issue is otherwise complete.Alerting tools drop issue refs from their descriptions (D4), pins re-…
update_alert_settings/update_mute_rule move reading guidance off tool…