tempdb Space fired on one collected sample and resolved on the next: the arm now holds for 3 consecutive collections behind the shared persistence gate, counted per collection rather than per sweep, with no-data freezing it instead of resolving to N/A (#3653 A5, tempdb slice)

[erikdarlingdata]

What does this PR do?

The tempdb Space alert fired on one collected sample and resolved on the next. It now has to hold for three consecutive collections to be an incident — the #3653 A5 persistence-gate rollout, the tempdb Space slice, on the shared AlertPersistenceGate that High CPU adopted in #3282. Partial for #3653 (the A5 line's tempdb arm; the Blocking Wait Time disjunction and the Long-Running Query opt-out are ruling-gated — Q4/Q5 on the issue — and untouched here). References #3653, does not close it.

The lie

AlertEngine.CheckTempDbSpaceAsync read the newest tempdb_stats row and fired the moment ReservedPercent >= TempDbSpaceThresholdPercent, then resolved on the first read under the bar. Measured on one production store class — 43 servers over 14 days — one server fired tempdb Space at or above the 80% bar 22 times, and every run was 1–4 breaching samples spanning at most 206 seconds before the percentage fell back. Twenty-two pages and twenty-two "tempdb Space Resolved" notices in two weeks about a tempdb that was never in trouble for longer than three and a half minutes. That is the exact shape #3282 measured on CPU and fixed with the persistence gate, and AlertPersistenceGate's own header says the built-in checks "adopt the same gate as #3282's own follow-up. One mechanism, not two." This is that follow-up for one arm.

Mechanism

The FIRE decision sits behind the gate; everything downstream is unchanged. CheckTempDbSpaceAsync computes breaching exactly as before, hands it to the gate, and fires only while the record says an incident is open — the threshold compare, the explicit Warning grade (#3660), the cooldown-governed standing-condition reminder, the card, the mute context and the NumericCurrentValue are byte-for-byte what they were. The resolve's falling edge now comes from the gate's Resolve outcome; with a clear count of 1 that is the same instant it always was.

Samples, not sweeps — and on this arm that is the whole fix. The alert sweep runs every 30 s; tempdb_stats lands a row once a minute at the shipped cadence (every five at the light tier). A gate that counted sweeps would reach K = 3 in 90 s of wall clock on ONE collected row — the measured 1–4-sample flap intact behind a gate that looked like it fixed it, which is the trap the CPU arm's own comment names. So TempDbSpaceInfo gains CollectionTimeUtc: the row's collection_time, which both SKUs' reads already ORDER BY … DESC LIMIT 1 — one projected column on a read that already exists, no new read, no new cadence, same one-row index seek (Darling TempDbSpaceSql, Lite LocalDataService.GetLatestTempDbSpaceAsync). The engine counts an observation only when that instant has advanced past the record's LastObservedSampleUtc; a null instant (the deprecated Dashboard's local read, every pre-existing fixture) degrades to counting sweeps — weaker, but the alert still fires, the same direction the CPU gate chose for the same null and for the same reason.

One mechanism, generalised rather than duplicated. The CPU arm's fresh-sample guard, AlertPersistenceGate.Evaluate call and skip-the-write-when-nothing-moved rule are lifted into ObservePersistenceAsync(records, metricName, key, sampleUtc, breaching, breachSamples, clearSamples) → (Outcome, IncidentOpen), and SaveCpuPersistenceAsync becomes SavePersistenceAsync(key, metricName, record). Both arms call them. The CPU arm's behaviour is unchanged — every #3282 pin in AlertEngineTests and BuiltinAlertPersistenceRungTests executes green here (below). A third consumer is a third record dictionary, a third seed line and a third pair of constants, nothing else.

The active flag is gone; the persisted record is the "incident open" bit. _activeTempDbSpaceAlert left the engine for exactly the reasons CPU's did: an in-memory flag re-announced a standing incident on the first post-restart sweep, and a bool beside the record would be a second source of truth for the same fact. The record is seeded at startup under the new metric name "tempdb Space" — a second row in the existing V118 (server_id, metric_name) persistence table, so no migration rung — and an already-open incident gets its cooldown clock stamped at seed, so the reminder waits a full cooldown rather than firing seconds after a restart. Same two-part guard as the CPU seed, same rationale, stated at the site.

No-data freezes the gate. The old arm fell through to its resolve branch on a null tempDb and announced "tempdb reserved space back to N/A" — a recovery nobody measured, rendered with the value it could not read. A null read is now neither a breach nor a clear; the streak and any open incident hold until a real row arrives. The "N/A" arm is deleted with it (the resolve is reached only with a non-null row by construction).

Why K = 3, and why the clear count is 1

TempDbSpaceBreachSamples = 3, derived from the measured flap: every one of the 22 runs was 1–4 samples; K = 3 keeps 2 of 22 — the two runs that held for three or more consecutive collections, which are the only two an operator could still have seen as TRUE when the message arrived. K = 2 would keep the two-sample runs (most of the noise); K = 4 drops both real ones and buys nothing the measurement asked for. Three collections is ~3 min at the one-minute default and ~15 min at the five-minute tier — the constant's comment says so. A constant and not a knob, for CpuBreachSamples' stated reasons (#3314's half-plumbed number is the outcome to avoid).

TempDbSpaceClearSamples = 1, and the comment says why it is NOT the CPU gate's 2: CpuClearSamples is 2 because #3282 measured the shape it guards — 99% then 23% one minute later inside a real saturation event, a mid-incident dip. The #3653 read counted breach RUNS and said nothing about dips inside a run; there is no evidence a tempdb resolve is ever contradicted by the next sample, and reserved tempdb is a slower gauge than scheduler CPU (it moves when allocation units are freed, not per ring-buffer tick). Picking 2 anyway would be the folklore constant the lineage rule refuses. If the fleet shows resolve/re-fire pairs inside one tempdb event, that is the number that moves, with the measurement beside it. Keeping 1 also means the resolve half of this arm is behaviourally unchanged.

Blast radius

• TempDbSpaceInfo gains a nullable property; the deprecated Dashboard's own TempDbSpaceInfo construction (DatabaseService.NocHealth.cs, frozen twin, no gate) is unaffected — null by default.
• DarlingAlertReadAdapter.TempDbSpaceSql and Lite's GetLatestTempDbSpaceAsync project one more column. DarlingAlertReadAdapterTests' dialect/table pins on the SQL still hold (no now(, no N', FROM tempdb_stats).
• AlertReadFailureSurfaceTests (Under store contention the alert pass fails first and silently, so the self-alert that would report it is the first thing lost #3013 census): AlertEngine.cs stays at 14 counted / 6 exempt. No catch block added or removed; the seed's new LoadAlertPersistenceAsync await has its readClock.Restart() before it, and the tempdb try body restarts the clock after the gate call. The one exemption key moved with the generalised save message ("Could not persist the {Metric} persistence gate"), so the census matches the one catch block that now serves both metrics.
• BuiltinAlertPersistenceRungTests' text pins on the CPU seed (if (cpuPersistence.Value.State.Firing), _lastCpuAlert[key] = _utcNow();) are untouched — the CPU seed block is byte-identical.
• Two Lite.Tests pins (LiteAlertForwardingTests.TempDb_ToastBody_MatchesTheOldLoop, TempDb_TheAzureCeiling_…) drove one sweep and asserted a fire; they now drive TempDbSpaceBreachSamples sweeps (their fixtures carry no instant, so they ride the pinned every-sweep-counts fallback). Assertions unchanged.
• Both READMEs' alert tables state the gate; the root README's TempDB row is pinned to the constants (TheReadmeAlertCatalogStatesTheTempDbSampleCountItActuallyUses), the same pin the High CPU row already carries.

What it does NOT do

• Does not close Brains-review campaign: deferred structural residue (from #3538 / #3539 / #3540 / #3541) #3653 — the A5 line stays unticked; tempdb is one of its three arms.
• Blocking Wait Time (duration-OR-consecutive disjunction) and Long-Running Query (opt-out coverage) are NOT touched: the blocking arm needs a single-snapshot bar nobody has measured and the LRQ arm needs a new knob — both are rulings (Q4/Q5) on the issue, not worker calls.
• No migration rung, no StorageVersion bump, no new knob, no CHANGELOG.md edit (entry handed to the coordinator).
• Does not change the tempdb threshold, the Warning grade, the reminder cadence, the card, or the resolve count (1, as today).
• Does not widen IAlertReadAdapter; the sample instant rides on the existing return type.

Test plan

Every touched project builds -c Release -p:EnableWindowsTargeting=true with 0 warnings from this change (PerformanceMonitor.Alerting, PerformanceMonitor.Darling.Service, Darling.Tests, PerformanceMonitorLite, Lite.Tests — the latter's one warning is pre-existing on dev in LogTailOverlapThresholdPinTests.cs:95, untouched).

The Darling.Tests engine pins were EXECUTED on this machine, not just compiled, through a throwaway net10.0 xunit harness outside the worktree (AssemblyName=Darling.Tests, referencing the built PerformanceMonitor.Alerting / Common / Notifications / Darling.Storage, compiling AlertEngineTests.cs, AlertPersistenceGateTests.cs, AlertReadFailureSurfaceTests.cs, BuiltinAlertPersistenceRungTests.cs, RepoFileAdoptionTests.cs, DocCommentHygieneTests.cs + helpers): 273 tests, 0 failed — the whole of AlertEngineTests (145, every #3282 CPU pin included), the gate's own 12, the #3013 census's 27 with the moved exemption key, the rung file's 10 (one Viewer-reflecting test excluded, it needs WPF), the LF-reader roster, and the cref/summary hygiene sweep over the tree.

Darling.Tests/AlertEngineTests, tempdb section — new (DriveTempDbAsync helper, the DriveCpuAsync twin):

• TempDb_OneOrTwoBreachingSamples_DoNotFire_TheThirdFiresExactlyOnce — K−1 → nothing; K → one fire; the cooldown reminder unchanged.
• TempDb_AStreakBrokenByOneClearSample_NeverFires_UntilThreeRunConsecutively — breach/clear/breach/breach → nothing; the fourth breach fires (the reset is a reset).
• TempDb_RepeatedCollectionTime_DoesNotAdvanceTheStreak — the same collection_time offered 3K times → nothing; two fresh rows then fire. The pin that matters most on this arm.
• TempDb_NoCollectionTime_CountsEverySweep — the documented fallback.
• TempDb_AMissingRowFreezesTheGate_AndNeverAnnouncesARecovery — null reads after a fire: no resolve, no re-announce; one real clear row resolves with the measured number.
• TempDb_PersistsTheStreakUnderItsOwnMetric_AndResumesAcrossARestart — the record is under "tempdb Space" with the streak and the instant; a new engine over the same store completes it.
• TempDb_ARestartDoesNotReAnnounceAnAlreadyOpenIncident — restart with Firing = true: no re-announce; the stamped clock is a cooldown, not a silence; resolves normally.
• TempDb_TheGateAdvancesUnderSuppression_SoOneUnsuppressedSampleDelivers.
• TempDb_APersistenceSaveFailure_StillFires_AndIsNotCountedAsASwallowedRead — through the shared save path.
• TempDbGateDefaults_AreDerivedFromTheMeasuredFlap — 3 / 1 / "tempdb Space".
• TheReadmeAlertCatalogStatesTheTempDbSampleCountItActuallyUses.

Re-shaped to drive the gate (assertions unchanged): TempDb_FiresAtThreshold_AndResolutionCarriesTheCurrentPercent, TempDb_TheAzureCeiling_SuppressesTheAlert_ThatTheAllocationWouldFire, TempDb_FiresAnExplicitWarning_AndNeverGradesCritical.

Lite.Tests/LiteAlertForwardingTests — the two tempdb pins drive K sweeps; compile-verified here, first executed in CI (their harness is the WPF App's statics, which this Mac cannot host). They are the same shape as TempDb_NoCollectionTime_CountsEverySweep, which executes green.

Which component(s) does this affect?

• Lite
• Darling
• Lite Tests
• Darling Tests
• SQL collection scripts
• Documentation
• Full Dashboard (deprecated)
• CLI Installer (deprecated)

How was this tested?

See the test plan: executed here via the harness (273/0), compile-verified on every touched project, CI is the first execution of the Lite.Tests pins. No SQL Server change — the collector and its tempdb_stats row are untouched; the change is read-side and engine-side.

Checklist

• I have read the contributing guide
• My code builds with zero warnings (dotnet build -c Debug)
• I have tested my changes against at least one SQL Server version (engine-level; the collected row shape is unchanged)
• I have not introduced any hardcoded credentials or server names