Repository navigation
Recs Stage B (severity): cap tuning-class CXPACKET, escalate meltdowns through THREADPOOL - #1494
Merged
Merged
Conversation
…tdowns through THREADPOOL Stage B change 4 -- the severity half -- on the shared FactScorer, applying identically to Lite, Dashboard, and Darling. - Cap the FINAL severity of tuning-class keys (CXPACKET/CXCONSUMER/QUERY_HIGH_DOP + all ANOMALY_*) at the 1.49 WARNING ceiling, released when an impact-bearing peer co-fires (THREADPOOL / SOS_SCHEDULER_YIELD / RESOURCE_SEMAPHORE). Caps the numeric Severity only; the band is derived downstream. - Raise THREADPOOL's CXPACKET amplifier +0.2 -> +0.5 and add a runnable-queue amplifier (+0.5) so a genuine parallelism -> thread-exhaustion meltdown reaches CRITICAL through THREADPOOL (an impact key, never capped). - Feed the runnable-queue amplifier from the EXISTING cpu_scheduler_stats snapshot's runnable_tasks_warning (SUM(runnable_tasks_count) >= cpu_count) via a RUNNABLE_TASKS context fact in the Lite + Darling fact collectors -- no new collector/table/migration. Dashboard fact collection untouched (amplifier no-ops). - Harden the Stage-A low-quality anomaly fallback to floor at 0.5 AFTER the confidence multiply (robust to a future sub-1.0 confidence). Tests: new FactScorerTests in Lite.Tests + Dashboard.Tests. Full suites green (Lite 1087, Dashboard 728, Darling 2066 + 133 live-PG skipped). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CRITICAL: the CXPACKET tuning-class severity cap's escape hatch released on sosPeer.BaseSeverity > 0 / rsPeer.BaseSeverity > 0 -- a near-zero bar. SOS_SCHEDULER_YIELD (0.75, null) and RESOURCE_SEMAPHORE (0.01, null) have no minimum-total guard in ScoreWaitFact, so BaseSeverity > 0 means only "the wait is present at all." SOS physically co-occurs with high CXPACKET (parallel workers yield -> SOS) and is emitted for any delta_wait_time_ms > 0, so a trivial SOS (e.g. 500ms/hr) released the cap on exactly the busy servers the cap targets, re-admitting the CXPACKET=CRITICAL noise the cap exists to kill. Gate SOS/RS on significance via the same HasSignificantWait helper the amplifiers use (SOS 0.25 = the CXPACKET SOS amplifier bar; RS 0.10 -- no amplifier bar exists, 10% of period is meaningful grant starvation). THREADPOOL keeps its presence check: its base is self-gating (>= 1hr total AND >= 1s avg), so BaseSeverity > 0 already means exhaustion. Corrected the now-false "belt-and-suspenders" comment. Regression tests added to BOTH Lite.Tests and Dashboard.Tests FactScorerTests: a trivial SOS or RESOURCE_SEMAPHORE below the bar leaves CXPACKET capped at 1.49 WARNING; a significant one releases it into CRITICAL. Verified failing against the old presence-only predicate. MINOR: the RUNNABLE_TASKS context-fact SQL (RunnableTaskStatsSql in Darling PgFactCollector.Resources.cs and the identical inline SQL in Lite DuckDbFactCollector.Resources.cs) filtered only collection_time <= end with ORDER BY collection_time DESC LIMIT 1. Add the lower bound (>= start, matching sibling CpuUtilizationSql) so a lapsed collection can't read an arbitrarily old snapshot from outside the analysis window. Kept the point-in-time LATEST snapshot rather than a MAX window-aggregate because runnable_tasks_warning is BOOLEAN and Postgres has no max(boolean) aggregate, so a byte-identical MAX() across the DuckDB and Postgres providers is not possible without a cast. SQL kept byte-identical between both providers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 12, 2026
pull Bot
pushed a commit
to ehtick/PerformanceMonitor
that referenced
this pull request
Jul 22, 2026
…schedule presets The viewer's Aggressive/Balanced/Low-Impact preset tables (CollectorSchedulePresets.Presets) were missing these three collectors in all three presets, so applying a preset in the Darling schedule editor left them at their code-default cadence instead of the preset's -- a Lite<->Darling drift flagged in the erikdarlingdata#1494 review. Each preset now carries the three at Lite's exact interval (Aggressive 2, Balanced 5, Low-Impact 15 min); retention is unchanged (it lives in the shared CollectorScheduleDefaults, which already carried all three, so only the preset-apply path had drifted). The class docstring promised a Darling.Tests pin against Lite's intervals, but no membership pin existed -- Presets_OnlyReferenceKnownCollectors only checks the reverse direction (no unknown collectors), which is why the gap went uncaught. Add Presets_CoverTheSamePinnedCollectorSet, mirroring Lite's own SchedulePresets_CoverTheSamePinnedCollectorSet, pinning the full 27-collector membership of every preset so this drift class fails a test next time. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pull Bot
pushed a commit
to ehtick/PerformanceMonitor
that referenced
this pull request
Jul 22, 2026
Restore three classifications the deprecated Dashboard's report.* / config.critical_issues SQL made but the ported shared analysis engine had dropped. All three are pure FactScorer additions over facts the collectors already emit (no collector/schema/install change), so they light up in Lite, the Dashboard, and Darling recommendations at once; each cites its source SQL line in a code comment. - Query Store disabled: INFO advisory folded into DB_CONFIG, detected as query_store_on_count < database_count (install/50:83). - Security-cache (TokenAndPermUserStore) growth: flat WARNING when the USERSTORE_TOKENPERM clerk >= 1 GB, off the context-only MEMORY_CLERKS fact (install/50:562,583). - Runnable-queue / worker pressure: standalone ScoreCpuFact arm on total_runnable_tasks_count tiered > 50/20/10 plus the small-box runnable_tasks_warning fallback (install/47:1839-1844); additive to and distinct from the erikdarlingdata#1494 THREADPOOL amplifier, which is unchanged. Advice wired for the new/expanded root keys (MEMORY_CLERKS + RUNNABLE_TASKS blocks; Query Store folded into ComposeDbConfig) so the cards are not contentless, sourced from the Dashboard's own message/recommendation text. FactScorer tests added in both Lite.Tests and Dashboard.Tests (tiers fire at the right band, stay silent below the bar, existing arms unchanged). The Lite clean-server seeder now defaults Query Store ON (the healthy value) so the QS-off arm stays dormant in fixtures not exercising it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
8 of 12 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage B change 4 — the severity half of the recs-engine remainder (the dedup half shipped in #1493; Stage A in #1491). All on the shared
FactScorer, so Lite, Dashboard, and Darling change identically. Blocking/deadlock detection is deliberately left on its current raw-count behavior.What changed
A. Tuning-class severity cap (
FactScorer.ScoreAll, new Layer 3). CXPACKET saturates its base severity to 1.0 at 25%-of-period, then its corroborating amplifiers (default CTFP, MAXDOP 0, high-DOP queries) multiplied it past the shared>= 1.5CRITICAL band on amplifier COUNT alone — so a pure parallelism-tuning situation e-mailed as CRITICAL. A final pass now caps the tuning-class keys (CXPACKET/CXCONSUMER/QUERY_HIGH_DOP+ everyANOMALY_*) at the 1.49 WARNING ceiling, with an escape hatch: released the moment an impact-bearing peer co-fires (THREADPOOL,SOS_SCHEDULER_YIELD, orRESOURCE_SEMAPHORE), because then the parallelism genuinely IS driving an outage. Caps the numericSeverityonly (the band is derived downstream) — noSeverityBandfork, parity preserved. Reality-check per the audit: CXPACKET is the only key with teeth today (ANOMALY_*+QUERY_HIGH_DOPalready max at 1.0); the rest is forward-safety.D. Impact escalation. So a genuine parallelism → worker-exhaustion meltdown still reaches CRITICAL,
THREADPOOL(an impact key, never capped) gets a heavier CXPACKET amplifier (+0.2 → +0.5) plus a new runnable-queue amplifier (+0.5). The runnable-queue signal is aRUNNABLE_TASKScontext fact read from the EXISTINGcpu_scheduler_statssnapshot'srunnable_tasks_warningflag (the collector's ownSUM(runnable_tasks_count) >= cpu_countheuristic) — no new collector, table, or migration. The fact is surfaced by the Lite + Darling fact collectors (Sourcecpu, carried into the amplifier lookup but never scored/rooted). The deprecated Dashboard's fact collection is left untouched — the amplifier simply no-ops there, and the cap + raised CXPACKET amplifier still escalate a Dashboard meltdown to CRITICAL through THREADPOOL. (cpu_scheduler_statsisn't collected on Azure SQL DB either, so the fact is absent and the amplifier no-ops there too.)B. Confidence-hardening. The Stage-A low-quality anomaly fallback now floors at 0.5 after the confidence multiply (
Math.Max(0.5, (…) * confidence)) instead of before — behaviorally identical today (confidence is a hardcoded 1.0 everywhere) but robust to a future sub-1.0 confidence that would otherwise silently drop a fired thin-baseline finding belowInferenceEngine's 0.5 root entry-point. Trustworthy z-path untouched.Runnable-tasks fact-flow
Both
DuckDbFactCollector(Lite) andPgFactCollector(Darling) read the latestcpu_scheduler_statsrow (total_runnable_tasks_count,runnable_tasks_warning) with byte-identical SQL and emit oneRUNNABLE_TASKSfact carryingrunnable_tasks_warningin metadata; the THREADPOOL runnable-queue amplifier predicates on that flag.Tests
New
FactScorerTestsin both Lite.Tests and Dashboard.Tests: CXPACKET alone caps to WARNING; CXPACKET + THREADPOOL releases the cap into CRITICAL; THREADPOOL reaches CRITICAL via the raised amplifier + the runnable-queue amplifier; the runnable-queue amplifier does not fire when the warning is clear; the confidence floor holds at 0.5 under a hypothetical sub-1.0 confidence.Full suites green: Lite.Tests 1087 passed, Dashboard.Tests 728 passed, Darling.Tests 2066 passed / 133 skipped (live-PG), 0 failures.
🤖 Generated with Claude Code