Skip to content

cpu_utilization stores each sample's UTC instant beside its server-local stamp and server_properties stores the engine's time-zone id beside its offset (Darling V134 / Lite v63), so UTC-window readers stop deriving an offset that is an hour wrong across DST (#3653 item 13, Q7 + Q8) - #3730

Merged
erikdarlingdata merged 2 commits into
devfrom
fix/3653-time-honesty-rung
Sep 19, 2026

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Partial #3653 — item 13 (Q7, Q8)

Darling V134 / Lite v63 — the "time honesty" rung. Two nullable columns that let the store say, for the first time, which clock two of its values are in: cpu_utilization_stats.sample_time_utc (the same instant sample_time records, in UTC) and server_properties.time_zone_id (the engine's own zone name, beside the utc_offset_minutes V16 added). Both SKUs, one PR, per the ruling's sequencing note.

The mechanism — two lies about time

cpu_utilization_stats.sample_time is the monitored server's LOCAL wall clock in a store whose every other timestamp is naive UTC. That is a documented convention, not an accident (CollectorTimestampFrameTests pins it): on the ring-buffer arm it is SYSDATETIME() minus each entry's age, Lite plots it in the server's own frame, and it is the collector's watermark. The cost of the convention is that every reader that aligns it against UTC has to derive the UTC value. Darling's #1262 de-skew recovers the offset per batch as MAX(sample_time) OVER (batch) − collection_time rounded to fifteen minutes; Lite shifts its whole window by the one utc_offset_minutes the store holds now. Both assume one offset per batch or per server, and across a DST transition that is false for exactly the rows nearest the change. Because the clock falls back, an older EDT-side sample carries a LATER local stamp than the newer EST-side one in the same poll, so the batch's MAX recovers −4 h and places the EST-side samples an hour early — silently, in the plausible direction. Lite's window applies today's offset to yesterday's samples and misses an hour's worth on the far side (the live test seeds exactly that shape and the pre-rung predicate returns [11] where the fixed one returns [22, 11]).

server_properties.utc_offset_minutes is an OFFSET — DATEDIFF(MINUTE, GETUTCDATE(), GETDATE()), the one in force at collection. #3231 already documented that subtracting it from a stored instant that can outlive a transition (get_index_usage's last_user_access, the PVS cleaner stamps) is exact on one side and an hour wrong on the other, and that nothing in the store can say which side a given instant was on. An offset cannot; a zone can, because AT TIME ZONE applies the rule in force at the instant.

What the rung does

  • Collector (CpuUtilizationCollector): every arm projects sample_time_utc LAST. Ring buffer: the identical two-step DATEADD (the [BUG] Lite - CPU Graph renders dots and no lines #2749 millisecond precision and the MemoryPressureEventsCollector: arithmetic overflow after #2751's millisecond-precision fix #2755 overflow split both apply) anchored on SYSUTCDATETIME() instead of SYSDATETIME() — both are runtime constants folded once per statement, so the pair differs by exactly the server's offset. Azure SQL DB: drs.end_time, which sample_time already reads; end_time is documented UTC and Azure's clock IS UTC, so the two frames are one value there and neither column lies. sample_time is not converted and the watermark stays on it: existing rows would be indistinguishable from converted ones, and a watermark that changed frame would re-ingest or skip one offset's worth of samples on the first poll after the upgrade.
  • Collector (ServerPropertiesCollector): time_zone_id = CURRENT_TIMEZONE_ID(), read in its own sp_executesql batch behind ProductMajorVersion >= 16 OR EngineEdition IN (5, 8) and inside TRY … CATCH. This is the one column in the file that genuinely needs a version gate and the comment says so rather than dressing it as the deferred-object-ref pattern: on SQL Server 2019 and earlier the function is a missing BUILT-IN, not a missing object, so a batch that names it fails to compile as a whole — OBJECT_ID() has nothing to test and a CASE cannot help. The dynamic batch compiles one level down, so a compile error there IS catchable. NULL where the engine cannot say; the main SELECT still reads no table and no function that could fail to compile (MainQuery_ReadsNoTable_… stays green).
  • Ladder: Migration(134, "time-honesty") — two ADD COLUMN IF NOT EXISTS, nullable, no default, no backfill; v_cpu_utilization_stats refreshed for the V14 reason (server_properties has no view). Lite v63: the same two ALTERs, non-fatal per statement, v_ views rebuilt on start. Generated CREATE TABLE carries both columns last on a fresh store. Viewer probe gains the V134 sentinel (cpu_utilization_stats.sample_time_utc, ordinal 109) and the top arm; V133's tests hand over the top-rung claims.
  • Why no backfill is the ONLY honest choice here, not merely the safe one: a pre-rung sample_time could be converted only by the offset its server had AT THAT SAMPLE'S INSTANT, and the store never recorded that — it holds the sparse on-load offset series and the per-batch estimate, which are exactly the DST-blind derivations this column replaces. A backfilled value would be the old lie written into the new column where a reader could no longer tell it from a measurement. And no zone can be recovered from an offset: several zones share every one.
  • MCP get_server_properties (both SKUs, byte-identical keys): utc_offset_minutes, time_zone_id, and time_zone_note — "time_zone_id is null: a pre-2022 engine (…), so only the offset in force at captured_at is known." The tool descriptions say the same.

Reader table — every cpu_utilization_stats reader on dev, and what it does now

Reader Decision Why
ViewerDataService.Cpu.CpuUtilizationSql (Darling viewer CPU tab + Overview lane) coalesce — COALESCE(sample_time_utc, <#1262 de-skew>) AS sample_time UTC-aligned plot through ForDisplay; post-rung rows placed by the stored instant, pre-rung rows exactly as before; window stays on collection_time
DarlingDataReader.CpuUtilizationSql (MCP get_cpu_utilization) coalesce — same projection, byte-identical UTC hours_back window; the payload field keeps its name, its value is UTC either way
LocalDataService.Cpu.GetCpuUtilizationAsync (Lite chart + MCP get_cpu_utilization) coalesce the WINDOW, keep the projection local — COALESCE(sample_time_utc, sample_time − to_minutes($offset)) BETWEEN utcStart AND utcEnd The window is a UTC question; for a NULL twin sample_time − offset ≥ utcStart is algebraically the old sample_time ≥ utcStart + offset, so the pre-rung population selects byte-for-byte as before. The projected SampleTime stays server-local because the chart plots the server's frame and the local stamp IS that frame with no offset applied
DarlingWorker.LatestCpuSql (CPU alert read) keep local sample_time here is the persistence gate's observation identity, compared by > against the value the same read stored last sweep. Switching it to the twin changes the frame exactly once, at the upgrade, so every server EAST of UTC would see its first post-rung sample read older than its last pre-rung one and the gate would freeze for one offset's worth of hours (#3282's trap). Doc says so; pinned
DarlingHealthReader.ServerSummaryCpuSql, DarlingFleetReader.FleetCpuSql, ViewerDataService.Overview.ServerSummaryCpuSql, Lite LocalDataService.Overview latest keep local — tiebreak only ORDER BY collection_time DESC, sample_time DESC carries no clock frame; within one batch both columns order identically outside a DST-straddling poll
PgAnomalyDetector.CpuWindowSql / HasBaselineDataSql, PgBaselineProvider CPU, PgDrillDownCollector.SpikePeakSql, PgFactCollector.Resources.CpuUtilizationSql, DailySummarySql cpu, FinOps ×3 (Darling); AnomalyDetector, BaselineProvider, DrillDownCollector, DuckDbFactCollector.Resources, DailySummary, FinOps ×4 (Lite); DarlingDataReader.CpuWindowAggregateSql / Lite GetCpuWindowAggregateAsync untouched — already honest Every one windows on collection_time, the collector prefix, which was always naive UTC; none reads sample_time at all. The brief's "PgAnomalyDetector.cs:151" and "alert engine High CPU window" turned out to be this shape (claim found not applicable — see the report)
MeasureCatalog Compose CPU measures deferred — fenced, and nothing to do Both CPU measures bucket on the collector's PrefixTimeColumnName = collection_time and never read sample_time; ComposeCompiler's make_interval(mins => utc_offset_minutes) shift is for annotation TimeColumns, not this table. If a sample-grain CPU measure keyed on sample_time is ever added, it should use COALESCE(sample_time_utc, …)
RdsCpuIngestor, DarlingMcpPgCpuUtilizationTools, FleetPgCpuSql, memory_pressure_events / memory_grant_stats readers not this table pg_cpu_utilization.sample_time is UTC from Performance Insights; memory_pressure_events.sample_time is UTC per #2932

What it does NOT do

It does not convert sample_time, rename it, or move the watermark. It does not backfill either column. It does not apply AT TIME ZONE anywhere yet — the zone is stored so a later lane can fix the #3231 de-skews at their read boundaries, and until then the offset does what it did, which is why the offset is kept rather than replaced (it is NULL-zone on most of today's fleet anyway). It does not touch MeasureCatalog (fenced; nothing needed), PgAnomalyDetector's window SQL (L1's, and it never read sample_time), the analysis windows, or any PgTarget* file. It adds no viewer surface for the zone: get_server_properties is the surface; the viewer's server tab has no properties grid without XAML surgery, so that is skipped and said here.

Evidence

  • SQL Server 2025, container at TZ=America/New_York (mine, torn down): the ring-buffer batch returns sample_time = 07:49:29.9864135, sample_time_utc = 11:49:29.9864135 — DISTINCT difference over every ring-buffer row is exactly 14400000 ms; server_properties returns utc_offset_minutes = -240, time_zone_id = 'Eastern Standard Time'. On a UTC container the difference is 0 ms and the zone UTC. Both Azure arms compile with the fourth column in place.
  • TimescaleDB 2.28.1 / PG 18 (DARLING_TEST_PG, mine, torn down): a store stopped at 133 (columns dropped, view narrowed) climbs to 134 in exactly one rung, then 0; both columns nullable, default-less, LAST; the refreshed view serves the twin; two rows through the real CpuUtilizationCollector.WritePayload over binary COPY in a fall-back-straddling batch plus two planted pre-rung rows — MCP and viewer reads return [pre1, pre2, straddle, newest] at their true UTC instants, and a control shows the de-skew alone puts the EST-side sample at newestUtc − 1 h; LatestCpuSql returns the local stamp; ServerPropertiesCollector.WritePayload writes -300 / 'Eastern Standard Time' and -240 / NULL, and get_server_properties publishes each with the right note.
  • DuckDB: a v62 database (columns dropped, v62 v_ views re-created OVER the narrower tables) climbs to 63 with the views present; both columns come back nullable at the tail; pre-rung rows read NULL through the re-expanded views; the Lite CPU window selects the post-DST row the offset-shifted window missed (red-first: old predicate [11], new [22, 11]).

Tests

New: Darling.Tests/TimeHonestyRungTests (rung SQL/doc pins, generated-last + COPY column, top-arm probe, both windowed reads' projection byte-identical, latest-row reads keep local + worker doc, properties read/payload, Lite twin pin; live class as above) and Lite.Tests/TimeHonestyRungTests (v63 block pins, generator trailing columns, MCP description parity, CPU window DST test, picker-range test, get_server_properties payload, v62→v63 upgrade). Extended: CpuUtilizationCollectorDefinitionTests (+4: twin arithmetic off SYSUTCDATETIME() and only there, Azure arms project end_time last, writer emits 4 values null-or-value), ServerPropertiesCollectorDefinitionTests (+2: gated dynamic batch shape, NULL zone stays NULL; 23 columns), CollectorTimestampFrameTests (+1: local clock feeds sample_time only, UTC clock feeds sample_time_utc only, identical arithmetic), ConsumedTimestampFrameDisciplineTests (census 67→68 / 49→50 / Utc 12→13, cpu_utilization_stats.sample_time_utc=Utc in the classified sequence), GoldenCollectorSchema (two appends), PgNumbackendsAndSampledMsRungTests (top-rung handoff; its live climb now stated against the ladder rather than as 1), PerfmonCounterTypeTests (v61 climb pinned to CurrentSchemaVersion), AgedDatabaseMigrationTests (named columns in the v47 fixture's INSERT — the first rung since v47 to widen server_properties — plus an assertion the v63 column landed).

Executed: whole Darling.Tests on this Mac with the rig — 11,157 total, 191 platform reds (WPF/SqlClient/Windows-path; dev baseline 188–192), zero lane-related; Lite harness 305 total, 301 green, the 4 reds the known IgnoredWaitTypes.Load → PresentationFramework artefact in wait-stats MCP tests untouched here. Every touched project builds at 0 warnings.

…cal stamp and server_properties stores the engine's time-zone id beside its offset (Darling V134 / Lite v63), so UTC-window readers stop deriving an offset that is an hour wrong across DST (#3653 item 13, Q7 + Q8)

Two lies about time, one rung. cpu_utilization_stats.sample_time is the
MONITORED SERVER'S LOCAL wall clock in a store whose every other timestamp
is naive UTC - a documented convention (Lite plots the server's frame; it
is the collector's watermark) - so every reader that aligned it against
UTC derived the offset: Darling's #1262 per-batch de-skew, Lite's
GetTimeRangeServerLocal shift by the one collected utc_offset_minutes.
Both place the samples nearest a DST transition an hour wrong, silently.
The collector now writes sample_time_utc beside the UNCHANGED local stamp
(the same two-step DATEADD off SYSUTCDATETIME(); drs.end_time on Azure,
where the clock is UTC anyway), and the windowed CPU readers - viewer,
MCP get_cpu_utilization, Lite's window - take COALESCE(sample_time_utc,
<old derivation>). Latest-row reads keep the local column: it is the CPU
alert gate's observation identity, and a frame change there would freeze
the gate for one offset's worth of hours east of UTC (#3282).

server_properties.utc_offset_minutes is the offset IN FORCE at collection
and cannot say which side of a transition an instant fell on (#3231); the
engine's own zone can. time_zone_id = CURRENT_TIMEZONE_ID(), read in its
own version/edition-gated sp_executesql batch inside TRY/CATCH because on
a pre-2022 engine it is a missing built-in and the batch that names it
fails to compile. NULL where the engine cannot say; get_server_properties
publishes the pair on both SKUs and says what a NULL zone means.

Nullable, no default, no backfill (the offset a row's server had at a
past instant is exactly what the store never recorded, and no zone can be
recovered from an offset); v_cpu_utilization_stats refreshed for the V14
reason; viewer probe gains the V134 sentinel and top arm; V133's tests
hand over the top-rung claims. Verified live: SQL Server 2025 at UTC-4
writes the two stamps exactly 14,400,000 ms apart and time_zone_id =
'Eastern Standard Time'; TimescaleDB 2.28.1 climbs 133 -> 134 and the
readers prefer the twin on a planted fall-back batch; DuckDB climbs
62 -> 63 with the v_ views present.

Partial #3653 - item 13 (Q7, Q8)
…load ordinal (sample_time_utc, V134 / v63) — the identity set is still the second result set after it (#3653 item 13)
@erikdarlingdata
erikdarlingdata force-pushed the fix/3653-time-honesty-rung branch from 6981fdf to d98145b Compare September 19, 2026 12:16
@erikdarlingdata
erikdarlingdata merged commit a8cf12e into dev Sep 19, 2026
6 of 8 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/3653-time-honesty-rung branch September 19, 2026 12:32
erikdarlingdata added a commit that referenced this pull request Sep 19, 2026
… adds two text[] columns to config_alert_settings whose DEFAULT is the production read's seeds, wired through the service seed/read, the Viewer's Settings window and select/upsert, the viewer probe's top arm, and both SKUs' get_alert_settings / update_alert_settings twins (#3653 A5, Q5) (#3736)

#3734 gave both SKUs the knob - program_name PREFIXES and exact logins
whose sessions the Long-Running Query alert does not evaluate, seeded
from the 7-day production read (SQLAgent - TSQL JobStep; NT AUTHORITY\
SYSTEM and NT AUTHORITY\NETWORK SERVICE) - editable on Lite, and on
Darling the seeds and nothing else, because a Darling alert knob has no
rung-free home: the store row is authoritative (ApplyToConfig replaces
config.Alerts wholesale), so a darling.json-only member is dead, and
McpAlertSettingsKeyTests pins both SKUs' get_alert_settings to one
shape, so neither could publish it until Darling could store it. This
is that store home, one rung after #3730's V134.

V135: long_running_query_excluded_program_name_prefixes and
long_running_query_excluded_logins, text[] NOT NULL, DEFAULT the two
seeded arrays - the one knob rung on this row that ships a non-empty
default, so a pre-rung row reads as exactly what DarlingAlertSettings
returned before the rung and what Lite ships, and the two SKUs and the
two sides of the rung evaluate one population. An operator's cleared
list is stored as an explicit '{}' and honoured: present-and-empty is
a decision, and nothing re-seeds it. AlertsConfig and the Viewer's
AlertSettingsRow initialise to the same seeds (pinned equal to the
rung's DEFAULT and to LongRunningQueryExclusions.Default*). No reload
beacon (V17's trigger), no GRANT (table-level), IF NOT EXISTS.

Plumbing, all at the appended ordinals: StoreConfigProvider seeds them
as $69/$70 and reads them at 67-68 (a column selected but not read
resets the knob on every worker start); the Viewer's select/upsert bind
them as $68/$69 and read 67-68, its Settings window gets two boxes
(AlertLrqExcluded*Box, so the #1840 wiring census sees them: gated on
the master switch, Restore Defaults puts the SEEDS back); the viewer
probe gains the V135 sentinel as the top arm (V134's top-rung claims
move here, as V134 moved V133's); DarlingAlertReader reads 67-68 for
get_alert_settings; update_alert_settings accepts
long_running_query.excluded_program_name_prefixes / .excluded_logins
through the shared normaliser (empty array = cleared arm); Lite's
get_alert_settings publishes the same two keys. Every description
states the two rules, the four classes, the seeds' grounding and why
the admin login is not a default.

TimeHonestyRungLivePostgresTests' simulated climb drops V135's columns
and stamp too and expects two rungs: the applier skips anything at or
below MAX(version), so a "one rung short" store must be short of the
TOP, which this rung now is.

Executed on this Mac: the real Darling.Tests.dll (WindowsDesktop stripped)
against PostgreSQL 18 in Docker - 402/402 across the rung, probe, seed/
read, viewer bind, MCP and engine classes (the seed -> read scratch
round-trip carries a cleared login list back empty and a padded prefix
list back normalised; the store's column DEFAULTs read back from the
catalogue as the seeds); full suite 11,451 with 191 platform reds
(PresentationFramework / SqlClient ConfigurationManager / Windows
paths), the same baseline as before this change.

Partial #3653 - item 6 (Q5). Does not close it.
erikdarlingdata added a commit that referenced this pull request Sep 20, 2026
…o a fall-back's repeated hour and the V134 local→UTC frame switch are new samples rather than an hour of silence; both SKUs' latest-CPU reads hand it sample_time_utc where the row has one (#3744) (#3774)

The persistence gate's freshness test was `sampleUtc > LastObservedSampleUtc`, fed the target's LOCAL sample_time. Every autumn the local clock repeats an hour, so every sample in that hour read as stale and the gate neither advanced a streak nor cleared an incident. The same `>` was why #3730 (V134) left the gate on the local stamp: switching to the UTC twin flips the frame once, and east of UTC the first UTC instant reads older than the last local one — a frozen gate for one offset's worth of hours on upgrade day.

Both are one defect: an identity compared by order across two clock frames. The gate now compares for EQUALITY (a different instant is a different sample whichever clock stamped it), which fixes the fall-back in either frame and makes the frame switch cost at most one extra count of one sample per server, once. No rung: config.alert_persistence_state needs no frame column when the frame is not load-bearing.

Darling's LatestCpuSql and Lite's overview read project sample_time_utc beside sample_time; both hosts hand the gate `twin ?? local`. The ORDER BYs stay on the local stamp on purpose (every row has it; an order on the twin would compare frames across batches). Docs on the snapshot, the record and the worker say which clock the value is in and why there is no frame field.

Fixtures: fall-back (01:59 → 01:00 advances the streak and clears an incident), east-of-UTC upgrade (fresh exactly once, re-read held, `!=` within the frame), west-of-UTC upgrade (counted once more at most), the #3282 repeated-instant pin unchanged; TimeHonestyRungTests' "keeps the local stamp" pin flips to "projects the twin, orders on the local"; the live PG class asserts ordinal 3 and the pre-rung NULL fallback; Lite twins for the engine fixtures, a source pin on the overview read and snapshot fold, and a DuckDB pin through GetServerSummaryAsync.
erikdarlingdata added a commit that referenced this pull request Sep 20, 2026
…e store has it, with the frame stated, so the autumn fall-back's repeated local hour lands instead of being dropped as already collected (#3778)

The ring-buffer arm dedups client-side against MAX(sample_time), a LOCAL wall-clock instant, and drops every
sample at or below it — which from the fall-back instant is every sample for an hour, on every non-UTC server,
once a year. The other half of #3744's blind hour; invisible because #3730's live straddle test planted through
WritePayload, below the dedup.

ICollectorDefinition gains UtcWatermarkColumn (base null; only cpu_utilization declares it, as sample_time_utc)
and CollectorContext gains WatermarkFromUtcColumn. Both hosts branch on the declaration: a definition without a
twin goes through the unchanged GetLastCollectedTimeAsync and its byte-identical SQL; a definition with one goes
through GetLastCollectedTimeWithFrameAsync, ONE statement reading MAX(twin), MAX(local), handing back the twin
in the UTC frame where any row carries it and the local maximum in the LOCAL frame otherwise. The collector
compares each row in the watermark's frame. The first post-upgrade run finds no twin and dedups local-to-local
exactly as before (zero duplicated, zero dropped beyond the old rule); every run after it is UTC-to-UTC.

Not the gate, not the de-skew readers, not sample_time's frame, not Azure (server-side filter on end_time,
UTC in both columns; query text and parameter pinned identical in either frame). No rung.

Pins: the three acceptance sequences through the real ReadAsync (fall-back hour lands under UTC and the local
control lands zero; the upgrade day at zero duplicates; Azure identical), the never-path kept-not-guessed, the
census (only cpu_utilization declares the twin, none beside a PerDatabaseWatermarkColumn); both hosts' SQL as
shipped strings and their branches as source; the DuckDB pair read on a real store; the PostgreSQL round trip
reading the runner's pair read at three moments; the #2797 IL gate pin tracking both readers.
erikdarlingdata added a commit that referenced this pull request Sep 20, 2026
…e store has it, with the frame stated, so the autumn fall-back's repeated local hour lands instead of being dropped as already collected (#3778) (#3789)

The ring-buffer arm dedups client-side against MAX(sample_time), a LOCAL wall-clock instant, and drops every
sample at or below it — which from the fall-back instant is every sample for an hour, on every non-UTC server,
once a year. The other half of #3744's blind hour; invisible because #3730's live straddle test planted through
WritePayload, below the dedup.

ICollectorDefinition gains UtcWatermarkColumn (base null; only cpu_utilization declares it, as sample_time_utc)
and CollectorContext gains WatermarkFromUtcColumn. Both hosts branch on the declaration: a definition without a
twin goes through the unchanged GetLastCollectedTimeAsync and its byte-identical SQL; a definition with one goes
through GetLastCollectedTimeWithFrameAsync, ONE statement reading MAX(twin), MAX(local), handing back the twin
in the UTC frame where any row carries it and the local maximum in the LOCAL frame otherwise. The collector
compares each row in the watermark's frame. The first post-upgrade run finds no twin and dedups local-to-local
exactly as before (zero duplicated, zero dropped beyond the old rule); every run after it is UTC-to-UTC.

Not the gate, not the de-skew readers, not sample_time's frame, not Azure (server-side filter on end_time,
UTC in both columns; query text and parameter pinned identical in either frame). No rung.

Pins: the three acceptance sequences through the real ReadAsync (fall-back hour lands under UTC and the local
control lands zero; the upgrade day at zero duplicates; Azure identical), the never-path kept-not-guessed, the
census (only cpu_utilization declares the twin, none beside a PerDatabaseWatermarkColumn); both hosts' SQL as
shipped strings and their branches as source; the DuckDB pair read on a real store; the PostgreSQL round trip
reading the runner's pair read at three moments; the #2797 IL gate pin tracking both readers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant