Skip to content

get_analysis_facts runs the anomaly detector on both engines, so the facts read stops omitting the detector's ANOMALY_* facts (#3691) - #3714

Merged
erikdarlingdata merged 1 commit into
devfrom
fix/3691-facts-include-anomalies
Sep 19, 2026
Merged

erikdarlingdata merged 1 commit into
devfrom
fix/3691-facts-include-anomalies

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

What does this PR do?

get_analysis_facts now runs the anomaly detector on both SKUs, so the facts read shows the detector's ANOMALY_* / ANOMALY_PG_* facts — scored, with the gate metadata they were scored on — instead of a fact set that silently omitted the one family whose metadata an operator most often wants to read. Twin edit: Lite/Analysis/AnalysisService.CollectAndScoreFactsAsync and Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.CollectAndScoreFactsAsync, same shape.

PARTIAL for #3691. Checklist line completed: get_analysis_facts never runs the anomaly detector on EITHER engine (DarlingAnalysisService.CollectAndScoreFactsAsync = collector + scorer; Lite twin) — anomaly lineage/gate metadata is visible only in findings. Pre-existing; twin edit.

The lie

The tool's description said "every observation the engine sees", and named ANOMALY_* facts explicitly ("For ANOMALY_* facts the metadata carries baseline_confidence…"). The read behind it was collector + scorer only; the detectors ran in the full pass and nowhere else. So on both engines the facts read never contained an anomaly fact, and the gate metadata lane 9 and #3584 wrote (deviation_sigma against fire_threshold, baseline_samples, baseline_tier, baseline_low_quality, threshold_lineage) was reachable only through a finding that had already crossed the severity floor. An anomaly that fired and scored under 0.5 was invisible everywhere; an operator asking "what did the detector see" got facts that did not include the detector's. PgTargetAnomalyTests' live e2e pinned the defect in so many words (Assert.Empty on the anomaly-source facts read, with a comment telling the reader not to look there).

The change

In CollectAndScoreFactsAsync, after collection and before scoring, the same detector the pass would run is run over the same AnalysisContext, and its facts are appended so they arrive scored with their metadata intact — collector → detector → scorer, the pass's order. On Darling the detector is the RESOLVED engine's (engine.Detector, off the one ResolveEngineAsync the read already performs): a PostgreSQL target gets PgTargetAnomalyDetector's ANOMALY_PG_* facts, a SQL Server target gets PgAnomalyDetector's. Nothing is constructed by hand. On Lite it is _anomalyDetector, the service's own.

The detector is gated exactly as the pass gates it: it does not run over a window the collector never observed (context.ObservedDurationMs > 0). A deviation is measured against the window's own rate and an unobserved window has none; running the detector there would hand the tools anomaly facts to render under an envelope whose prose says nothing windowed exists. So both tools' unobserved envelope keeps describing exactly the point-in-time facts it names, byte for byte. A collector that observed the window and emitted no fact does NOT short-circuit past the detector (the #3653 rule the pass follows): the empty-set early return now sits after the detector's say.

The tool's baseline_confidence projection (an anomaly-source fact's confidence key renamed at read time, #3538 A6) now has something to project; it is unchanged.

Cost honesty

The read now pays the detector's reads on top of the collector's — the same reads the full pass already pays for the same answer: HasBaselineDataAsync plus one window read and one baseline lookup per detector family. PostgreSQL target: 5 detector families. SQL Server target (either SKU): 9 detector families. The baseline lookups go through the per-server baseline cache the pass populates, so a facts read shortly after a pass mostly hits it. No include_anomalies flag existed and none is added: the description says this is the pass's fact set, so it is. This is an on-demand MCP path, not the alert path.

audit_config's SQL Server arm also reads through CollectAndScoreFactsAsync (1-hour window) to pick out CONFIG_* facts and therefore pays the detector too. Disclosed rather than fenced: fencing would mean a service-level switch for one incidental caller, and the config facts it reads are unaffected. The coordinator can ask for a fence if the measured cost warrants it.

What it does NOT do

  • Does not touch any PgTarget* production file, detector or baseline provider; does not change what any detector emits or how the scorer grades it.
  • Does not change the full pass (AnalyzeAsync) — the AnalysisWindowEmptyRuleParityTests pins over it are untouched and green.
  • Does not change the SQL Server target's pass output; the only behavioural change is on the facts read, and it is additive (anomaly facts appear beside the collector's, never in place of them).
  • Does not mirror to deprecated/Dashboard (its AnalysisService.CollectAndScoreFactsAsync has the same shape); listed for the coordinator, not done here — outside this lane's declared boundary and the frozen twin is on bug-fix support only.
  • Does not address the v3 notification-ceiling item on PostgreSQL-target analysis engine: v2/v3 slices and the v1 lanes' residue (from #3542) #3691.

Descriptions

Both SKUs' get_analysis_facts descriptions gain one shared sentence (before the pinned baseline_confidence disambiguation): the detector runs on this read, which metadata the anomaly facts carry, that sub-floor fires are included, and what it costs. The sentence is added to McpMissMessageParityPinTests.SharedMissFragments so it cannot drift between SKUs. Both instructions files: the tool row now reads "collect+detect+score … the anomaly detector's ANOMALY_* facts included", and the as_of paragraph no longer reserves anomaly detection for analyze_server — the anchor reaches both tools' detection now, which is also what the anchored-window prose already promised.

Tests

New / changed — all compile-verified here; execution noted per item:

  • Darling/Darling.Tests/AnalysisFactsReadRunsDetectorParityTests.cs (NEW, 3 tests) — source pins over BOTH services' CollectAndScoreFactsAsync bodies (the AnalysisWindowEmptyRuleParityTests pattern; no CI test project references both SKU assemblies): collector < witness gate < detector < append < empty-return < scorer; exactly one detector call and one scoring call; the gate is > 0 (pass polarity) and no <= 0 spelling; no detector constructed inside the read; Darling uses engine.Detector off ResolveEngineAsync, Lite uses _anomalyDetector; both tool descriptions carry the new sentence on the get_analysis_facts attribute specifically, and both instructions files carry the new row text and have lost the old reservation. Executed on this Mac: 3/3 green; red-first proven by rebuilding against dev's DarlingAnalysisService.cs (2/3 red, as designed — the description pin stays green because it reads the tool files).
  • Darling/Darling.Tests/PgTargetAnomalyTests.cs (CHANGED, gated live e2e) — the Assert.Empty pin of the defect becomes the exit criterion: the lane-9 planted TPS spike → get_analysis_facts filtered to the anomaly source returns exactly the detector's key set (ANOMALY_PG_TPS, _SESSION_SPIKE, _CPU_SPIKE, _DEADLOCK_RATE), and the TPS fact carries value 60.0, a positive severity, deviation_sigma = display cap, fire_threshold = the modified-z bar, baseline_samples > 0, threshold_lineage = 0, baseline_confidence = 1.0 and NO raw confidence key. Executed here against a throwaway timescale/timescaledb:2.28.1-pg18 (Docker context desktop-linux, torn down): 39/39 green; red-first: Assert.NotEmpty() Failure: Collection was empty on dev's service. Also executed green on the same container: AnalysisCoverageLivePostgresTests, AnalysisAsOfAnchorTests, AnalyzeWindowEmptyLivePostgresTests, DarlingAnalysisPipelineTests, PgTargetFactCollectorTests, DarlingAnalysisServiceEngineRoutingTests, PgTargetMcpSurfaceTests, FactSourceRegistryTests, AnalysisPassCommandTimeoutTests, RepoFileAdoptionTests, DocCommentHygieneTests (77/77, run from inside the repo tree).
  • Lite.Tests/AnomalyDetectorTests.cs (CHANGED, +2 tests) — CollectAndScoreFacts_RunsTheDetector_SoTheSpikeIsAScoredFactOnTheFactsRead: the existing batch-request spike fixture plus a wait_stats series for the coverage witness, driven through the real AnalysisService.CollectAndScoreFactsAsync (anchored by name on the class's _analysisEnd, so the baseline bucket is the seeded hour): ANOMALY_BATCH_REQUESTS is present, source anomaly, severity > 0, with deviation_sigma/fire_threshold/baseline_samples/baseline_tier/baseline_low_quality/confidence intact, and the collector's own SOS_SCHEDULER_YIELD fact still beside it. CollectAndScoreFacts_DoesNotRunTheDetector_OverAnUnobservedWindow: same spike seeds without the witness → unobserved coverage, no ANOMALY_* fact. First executed in CI — AnalysisService lives in the WPF project and cannot load on this Mac. Right by construction: the seeds are the spike test's verbatim, the witness series is the 15-minute cadence AnalysisCoverageTests proves observed, and the scorer's deviation arm returns ≥ 0.5 × confidence at the fire threshold.
  • Lite.Tests/McpMissMessageParityPinTests.cs (CHANGED) — the new shared description sentence joins SharedMissFragments. Executed here in a net10.0 harness (the class reads source only): 61/61 green.

Census pre-checks: AnalysisWindowEmptyRuleParityTests (pass-body slice ends at the first CollectAndScoreFactsAsync( — no new mention added inside the pass), DarlingAnalysisServiceEngineRoutingTests (still exactly 3 await ResolveEngineAsync(; resolve precedes collect in the read), PgTargetMcpSurfaceTests (audit_config arm order untouched), FactSourceRegistryTests (source description untouched), McpPayloadContractCensusTests (total_facts = facts.Count untouched), DocCommentHygieneTests, RepoFileAdoptionTests (new file uses the non-LF reader with single-line anchors — no roster entry owed). No PgTarget* production file, detector, baseline provider, FactScorer, InferenceEngine or root partial touched.

Which component(s) does this affect?

  • Lite
  • Darling
  • Lite Tests
  • Darling Tests
  • SQL collection scripts
  • Documentation
  • Full Dashboard (deprecated)
  • CLI Installer (deprecated)

How was this tested?

Builds (zero new warnings): PerformanceMonitor.Darling.Service, PerformanceMonitorLite (-p:EnableWindowsTargeting=true), Darling.Tests, Lite.Tests (one pre-existing xUnit2000 warning in LogTailOverlapThresholdPinTests, untouched), deprecated/Dashboard.Tests. Live PostgreSQL execution as above against PG18/TimescaleDB 2.28.1 in a throwaway container. Lite behavioural tests are CI-first.

Checklist

  • I have read the contributing guide
  • My code builds with zero warnings (dotnet build -c Debug)
  • I have tested my changes against at least one SQL Server version (PostgreSQL-target path executed live; SQL Server path is the Lite fixture, CI-first)
  • I have not introduced any hardcoded credentials or server names

…, so the facts read shows the detector's ANOMALY_* facts (#3691)
@erikdarlingdata
erikdarlingdata merged commit d49650f into dev Sep 19, 2026
7 of 8 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/3691-facts-include-anomalies branch September 19, 2026 10:43
erikdarlingdata added a commit that referenced this pull request Sep 19, 2026
…aly facts-read pin takes the measured verdict (1); two co-fire boost summaries stop claiming the fact carries 0
erikdarlingdata added a commit that referenced this pull request Sep 19, 2026
…the 2026-09-19 read and threshold_lineage flips to 1, so a validated bar stops calling itself a guess (#3691 lineage flip) (#3723)

* PostgreSQL bars the fleet measured now say so: lineage comments and threshold_lineage flipped to measured (#3691 lineage flip)

The 2026-09-19 calibration read 50 Aurora PostgreSQL clusters of the dogfood fleet (14 d counters, 7 d
sessions/statements/waits). Every v1 bar it validated where it stood - temp 1/10 MiB/s, backlog 3/10x/5,
deadlocks 5/20 (PG-side re-read), bad-actor busy floor 0.05 (+ shares measured-conditional), wait rollups
0.20/1.0 and io 0.40/2.0 (Aurora), CPU 80/95 (Aurora Serverless v2), sessions 0.8/0.9 (engine-defined ceiling,
bands measured against it), TPS floor 50 / fallback 500, CPU floor 40 / fallback 95, deadlock floor 1/h, wait
fallback 500 ms/s - carries a measured comment naming its percentile, population and date, and the facts graded
only on measured or engine-defined bars stamp threshold_lineage = 1. Still unmeasured and still saying so: wait
standout bars and sampling floor, session COUNT floors, the ratio families' firing multiple, every co-fire
boost, checkpoint (not applicable on Aurora), buffer cache. No bar value changed - the diff touches no const
declaration; the only code lines are the stamps and one bool the detector passes to its z-metadata helper.

PgTargetMeasuredLineageTests pins each flipped constant's comment block to the date and population, pins the
unmeasured ones to still say so, and pins the flag per file.

* Rebase onto dev (#3714 get_analysis_facts anomaly read): its TPS-anomaly facts-read pin takes the measured verdict (1); two co-fire boost summaries stop claiming the fact carries 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant