You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
The CPU collector's watermark is the target's LOCAL sample_time and its dedup drops every sample at or below it, so the autumn fall-back hour never reaches the store — the other half of #3744's blind hour #3778
#3744 made the CPU alert gate see the repeated hour after an autumn fall-back as new samples (equality, not order — PR #3774). It does not make CPU alerting see that hour end to end, because a second mechanism upstream of the gate discards the hour before any row reaches the store: the CPU collector's watermark is the target's LOCAL sample_time, and its ring-buffer read drops every sample at or below it.
PerformanceMonitor.Collectors/CpuUtilizationCollector.cs:220 — public override string? WatermarkColumn => "sample_time"; — so both hosts' watermark reads (DarlingCollectorRunner.GetLastCollectedTimeAsync, Lite RemoteCollectorService.GetLastCollectedTimeAsync) hand the collector MAX(sample_time) of the rows already stored: a local wall-clock instant.
CpuUtilizationCollector.cs:273 — if (!context.Target.IsAzureSqlDb && context.Watermark.HasValue && sampleTime <= context.Watermark.Value) — the ring-buffer arm's client-side dedup: a sample whose LOCAL stamp is at or below the watermark is dropped as already collected. (Azure's arm binds the same watermark server-side as @last_sample_time, :105, :251-256; end_time there is already UTC, so the Azure arm is not affected.)
From the fall-back instant, the local clock repeats an hour. Every ring-buffer sample for the next hour carries a stamp ≤ the last one stored before the transition, so the collector discards all of them: no cpu_utilization_stats row lands for roughly one hour on every non-UTC server, once a year — the chart has a gap, the windowed reads have a hole, and the alert gate is blind for that hour because there is nothing to see. The gate fix (#3744) is the necessary half; this is the other half, and it was invisible because #3730's live straddle test planted its batch through WritePayload, bypassing the dedup, and the issue's premise ("every new sample for the next hour carries a stamp ≤ the last recorded") is true only of samples that land — and today none do.
Why it is not a one-line switch to sample_time_utc
V134 (#3730) added sample_time_utc beside the local stamp and left the watermark alone deliberately, for the same upgrade-day reason it left the gate's identity: a watermark that changes frame at the upgrade either re-ingests or skips one offset's worth of samples on every server, once. Concretely: MAX(sample_time_utc) is NULL on a store with no post-rung rows, so the first post-upgrade run would take the first-run fallback window and re-collect up to ~60 samples the store already holds under their local stamps — a one-time duplicate hour per server — unless the dedup also knows the frame of the rows it is comparing against. The fix has to carry that story: e.g. watermark on COALESCE(MAX(sample_time_utc), <derived from MAX(sample_time) and the batch offset>), or a two-column watermark with the frame stated, in the shared collector file (both SKUs) with the CollectorTimestampFrameTests / TimeHonestyRungTests pins moved with it. Whichever shape, the acceptance is the same: a fall-back sequence through the REAL ReadAsync dedup (not WritePayload) lands the repeated hour; an upgrade sequence neither drops nor duplicates more than one sample.
What this is NOT
Not the gate (fixed in #3744 / #3774). Not the de-skew readers (V134 covered them). Not Azure SQL DB (its arm is UTC already).
Found by the #3744 lane while verifying its fix (PR #3774, "What it does NOT do"). Descends from #3730 (Q7 of #3653) and #3744. Partial #3653 residue.
Activity
The CPU collector's watermark and dedup key are the UTC twin where th…
The CPU collector's watermark and dedup key are the UTC twin where th…
erikdarlingdata commentedon Sep 20, 2026
Closed by #3789 (merged to dev).
Wire the in-memory server watermark cache into DarlingCollectorRunner (…
Pin ServerWatermarkCache and the cache-eligibility defaults (#4197 pa…