Skip to content

The CPU collector's watermark is the target's LOCAL sample_time and its dedup drops every sample at or below it, so the autumn fall-back hour never reaches the store — the other half of #3744's blind hour #3778

Closed

Description

@erikdarlingdata

The lie

#3744 made the CPU alert gate see the repeated hour after an autumn fall-back as new samples (equality, not order — PR #3774). It does not make CPU alerting see that hour end to end, because a second mechanism upstream of the gate discards the hour before any row reaches the store: the CPU collector's watermark is the target's LOCAL sample_time, and its ring-buffer read drops every sample at or below it.

  • PerformanceMonitor.Collectors/CpuUtilizationCollector.cs:220 — public override string? WatermarkColumn => "sample_time"; — so both hosts' watermark reads (DarlingCollectorRunner.GetLastCollectedTimeAsync, Lite RemoteCollectorService.GetLastCollectedTimeAsync) hand the collector MAX(sample_time) of the rows already stored: a local wall-clock instant.
  • CpuUtilizationCollector.cs:273 — if (!context.Target.IsAzureSqlDb && context.Watermark.HasValue && sampleTime <= context.Watermark.Value) — the ring-buffer arm's client-side dedup: a sample whose LOCAL stamp is at or below the watermark is dropped as already collected. (Azure's arm binds the same watermark server-side as @last_sample_time, :105, :251-256; end_time there is already UTC, so the Azure arm is not affected.)

From the fall-back instant, the local clock repeats an hour. Every ring-buffer sample for the next hour carries a stamp ≤ the last one stored before the transition, so the collector discards all of them: no cpu_utilization_stats row lands for roughly one hour on every non-UTC server, once a year — the chart has a gap, the windowed reads have a hole, and the alert gate is blind for that hour because there is nothing to see. The gate fix (#3744) is the necessary half; this is the other half, and it was invisible because #3730's live straddle test planted its batch through WritePayload, bypassing the dedup, and the issue's premise ("every new sample for the next hour carries a stamp ≤ the last recorded") is true only of samples that land — and today none do.

Why it is not a one-line switch to sample_time_utc

V134 (#3730) added sample_time_utc beside the local stamp and left the watermark alone deliberately, for the same upgrade-day reason it left the gate's identity: a watermark that changes frame at the upgrade either re-ingests or skips one offset's worth of samples on every server, once. Concretely: MAX(sample_time_utc) is NULL on a store with no post-rung rows, so the first post-upgrade run would take the first-run fallback window and re-collect up to ~60 samples the store already holds under their local stamps — a one-time duplicate hour per server — unless the dedup also knows the frame of the rows it is comparing against. The fix has to carry that story: e.g. watermark on COALESCE(MAX(sample_time_utc), <derived from MAX(sample_time) and the batch offset>), or a two-column watermark with the frame stated, in the shared collector file (both SKUs) with the CollectorTimestampFrameTests / TimeHonestyRungTests pins moved with it. Whichever shape, the acceptance is the same: a fall-back sequence through the REAL ReadAsync dedup (not WritePayload) lands the repeated hour; an upgrade sequence neither drops nor duplicates more than one sample.

What this is NOT

Not the gate (fixed in #3744 / #3774). Not the de-skew readers (V134 covered them). Not Azure SQL DB (its arm is UTC already).

Found by the #3744 lane while verifying its fix (PR #3774, "What it does NOT do"). Descends from #3730 (Q7 of #3653) and #3744. Partial #3653 residue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions