Repository navigation
cpu_utilization stores each sample's UTC instant beside its server-local stamp and server_properties stores the engine's time-zone id beside its offset (Darling V134 / Lite v63), so UTC-window readers stop deriving an offset that is an hour wrong across DST (#3653 item 13, Q7 + Q8) - #3730
Merged
Merged
Conversation
…cal stamp and server_properties stores the engine's time-zone id beside its offset (Darling V134 / Lite v63), so UTC-window readers stop deriving an offset that is an hour wrong across DST (#3653 item 13, Q7 + Q8) Two lies about time, one rung. cpu_utilization_stats.sample_time is the MONITORED SERVER'S LOCAL wall clock in a store whose every other timestamp is naive UTC - a documented convention (Lite plots the server's frame; it is the collector's watermark) - so every reader that aligned it against UTC derived the offset: Darling's #1262 per-batch de-skew, Lite's GetTimeRangeServerLocal shift by the one collected utc_offset_minutes. Both place the samples nearest a DST transition an hour wrong, silently. The collector now writes sample_time_utc beside the UNCHANGED local stamp (the same two-step DATEADD off SYSUTCDATETIME(); drs.end_time on Azure, where the clock is UTC anyway), and the windowed CPU readers - viewer, MCP get_cpu_utilization, Lite's window - take COALESCE(sample_time_utc, <old derivation>). Latest-row reads keep the local column: it is the CPU alert gate's observation identity, and a frame change there would freeze the gate for one offset's worth of hours east of UTC (#3282). server_properties.utc_offset_minutes is the offset IN FORCE at collection and cannot say which side of a transition an instant fell on (#3231); the engine's own zone can. time_zone_id = CURRENT_TIMEZONE_ID(), read in its own version/edition-gated sp_executesql batch inside TRY/CATCH because on a pre-2022 engine it is a missing built-in and the batch that names it fails to compile. NULL where the engine cannot say; get_server_properties publishes the pair on both SKUs and says what a NULL zone means. Nullable, no default, no backfill (the offset a row's server had at a past instant is exactly what the store never recorded, and no zone can be recovered from an offset); v_cpu_utilization_stats refreshed for the V14 reason; viewer probe gains the V134 sentinel and top arm; V133's tests hand over the top-rung claims. Verified live: SQL Server 2025 at UTC-4 writes the two stamps exactly 14,400,000 ms apart and time_zone_id = 'Eastern Standard Time'; TimescaleDB 2.28.1 climbs 133 -> 134 and the readers prefer the twin on a planted fall-back batch; DuckDB climbs 62 -> 63 with the v_ views present. Partial #3653 - item 13 (Q7, Q8)
…load ordinal (sample_time_utc, V134 / v63) — the identity set is still the second result set after it (#3653 item 13)
erikdarlingdata
force-pushed
the
fix/3653-time-honesty-rung
branch
from
September 19, 2026 12:16
6981fdf to
d98145b
Compare
This was referenced Sep 19, 2026
Closed
erikdarlingdata
added a commit
that referenced
this pull request
Sep 19, 2026
… adds two text[] columns to config_alert_settings whose DEFAULT is the production read's seeds, wired through the service seed/read, the Viewer's Settings window and select/upsert, the viewer probe's top arm, and both SKUs' get_alert_settings / update_alert_settings twins (#3653 A5, Q5) (#3736) #3734 gave both SKUs the knob - program_name PREFIXES and exact logins whose sessions the Long-Running Query alert does not evaluate, seeded from the 7-day production read (SQLAgent - TSQL JobStep; NT AUTHORITY\ SYSTEM and NT AUTHORITY\NETWORK SERVICE) - editable on Lite, and on Darling the seeds and nothing else, because a Darling alert knob has no rung-free home: the store row is authoritative (ApplyToConfig replaces config.Alerts wholesale), so a darling.json-only member is dead, and McpAlertSettingsKeyTests pins both SKUs' get_alert_settings to one shape, so neither could publish it until Darling could store it. This is that store home, one rung after #3730's V134. V135: long_running_query_excluded_program_name_prefixes and long_running_query_excluded_logins, text[] NOT NULL, DEFAULT the two seeded arrays - the one knob rung on this row that ships a non-empty default, so a pre-rung row reads as exactly what DarlingAlertSettings returned before the rung and what Lite ships, and the two SKUs and the two sides of the rung evaluate one population. An operator's cleared list is stored as an explicit '{}' and honoured: present-and-empty is a decision, and nothing re-seeds it. AlertsConfig and the Viewer's AlertSettingsRow initialise to the same seeds (pinned equal to the rung's DEFAULT and to LongRunningQueryExclusions.Default*). No reload beacon (V17's trigger), no GRANT (table-level), IF NOT EXISTS. Plumbing, all at the appended ordinals: StoreConfigProvider seeds them as $69/$70 and reads them at 67-68 (a column selected but not read resets the knob on every worker start); the Viewer's select/upsert bind them as $68/$69 and read 67-68, its Settings window gets two boxes (AlertLrqExcluded*Box, so the #1840 wiring census sees them: gated on the master switch, Restore Defaults puts the SEEDS back); the viewer probe gains the V135 sentinel as the top arm (V134's top-rung claims move here, as V134 moved V133's); DarlingAlertReader reads 67-68 for get_alert_settings; update_alert_settings accepts long_running_query.excluded_program_name_prefixes / .excluded_logins through the shared normaliser (empty array = cleared arm); Lite's get_alert_settings publishes the same two keys. Every description states the two rules, the four classes, the seeds' grounding and why the admin login is not a default. TimeHonestyRungLivePostgresTests' simulated climb drops V135's columns and stamp too and expects two rungs: the applier skips anything at or below MAX(version), so a "one rung short" store must be short of the TOP, which this rung now is. Executed on this Mac: the real Darling.Tests.dll (WindowsDesktop stripped) against PostgreSQL 18 in Docker - 402/402 across the rung, probe, seed/ read, viewer bind, MCP and engine classes (the seed -> read scratch round-trip carries a cleared login list back empty and a padded prefix list back normalised; the store's column DEFAULTs read back from the catalogue as the seeds); full suite 11,451 with 191 platform reds (PresentationFramework / SqlClient ConfigurationManager / Windows paths), the same baseline as before this change. Partial #3653 - item 6 (Q5). Does not close it.
This was referenced Sep 19, 2026
erikdarlingdata
added a commit
that referenced
this pull request
Sep 20, 2026
…o a fall-back's repeated hour and the V134 local→UTC frame switch are new samples rather than an hour of silence; both SKUs' latest-CPU reads hand it sample_time_utc where the row has one (#3744) (#3774) The persistence gate's freshness test was `sampleUtc > LastObservedSampleUtc`, fed the target's LOCAL sample_time. Every autumn the local clock repeats an hour, so every sample in that hour read as stale and the gate neither advanced a streak nor cleared an incident. The same `>` was why #3730 (V134) left the gate on the local stamp: switching to the UTC twin flips the frame once, and east of UTC the first UTC instant reads older than the last local one — a frozen gate for one offset's worth of hours on upgrade day. Both are one defect: an identity compared by order across two clock frames. The gate now compares for EQUALITY (a different instant is a different sample whichever clock stamped it), which fixes the fall-back in either frame and makes the frame switch cost at most one extra count of one sample per server, once. No rung: config.alert_persistence_state needs no frame column when the frame is not load-bearing. Darling's LatestCpuSql and Lite's overview read project sample_time_utc beside sample_time; both hosts hand the gate `twin ?? local`. The ORDER BYs stay on the local stamp on purpose (every row has it; an order on the twin would compare frames across batches). Docs on the snapshot, the record and the worker say which clock the value is in and why there is no frame field. Fixtures: fall-back (01:59 → 01:00 advances the streak and clears an incident), east-of-UTC upgrade (fresh exactly once, re-read held, `!=` within the frame), west-of-UTC upgrade (counted once more at most), the #3282 repeated-instant pin unchanged; TimeHonestyRungTests' "keeps the local stamp" pin flips to "projects the twin, orders on the local"; the live PG class asserts ordinal 3 and the pre-rung NULL fallback; Lite twins for the engine fixtures, a source pin on the overview read and snapshot fold, and a DuckDB pin through GetServerSummaryAsync.
erikdarlingdata
added a commit
that referenced
this pull request
Sep 20, 2026
…e store has it, with the frame stated, so the autumn fall-back's repeated local hour lands instead of being dropped as already collected (#3778) The ring-buffer arm dedups client-side against MAX(sample_time), a LOCAL wall-clock instant, and drops every sample at or below it — which from the fall-back instant is every sample for an hour, on every non-UTC server, once a year. The other half of #3744's blind hour; invisible because #3730's live straddle test planted through WritePayload, below the dedup. ICollectorDefinition gains UtcWatermarkColumn (base null; only cpu_utilization declares it, as sample_time_utc) and CollectorContext gains WatermarkFromUtcColumn. Both hosts branch on the declaration: a definition without a twin goes through the unchanged GetLastCollectedTimeAsync and its byte-identical SQL; a definition with one goes through GetLastCollectedTimeWithFrameAsync, ONE statement reading MAX(twin), MAX(local), handing back the twin in the UTC frame where any row carries it and the local maximum in the LOCAL frame otherwise. The collector compares each row in the watermark's frame. The first post-upgrade run finds no twin and dedups local-to-local exactly as before (zero duplicated, zero dropped beyond the old rule); every run after it is UTC-to-UTC. Not the gate, not the de-skew readers, not sample_time's frame, not Azure (server-side filter on end_time, UTC in both columns; query text and parameter pinned identical in either frame). No rung. Pins: the three acceptance sequences through the real ReadAsync (fall-back hour lands under UTC and the local control lands zero; the upgrade day at zero duplicates; Azure identical), the never-path kept-not-guessed, the census (only cpu_utilization declares the twin, none beside a PerDatabaseWatermarkColumn); both hosts' SQL as shipped strings and their branches as source; the DuckDB pair read on a real store; the PostgreSQL round trip reading the runner's pair read at three moments; the #2797 IL gate pin tracking both readers.
erikdarlingdata
added a commit
that referenced
this pull request
Sep 20, 2026
…e store has it, with the frame stated, so the autumn fall-back's repeated local hour lands instead of being dropped as already collected (#3778) (#3789) The ring-buffer arm dedups client-side against MAX(sample_time), a LOCAL wall-clock instant, and drops every sample at or below it — which from the fall-back instant is every sample for an hour, on every non-UTC server, once a year. The other half of #3744's blind hour; invisible because #3730's live straddle test planted through WritePayload, below the dedup. ICollectorDefinition gains UtcWatermarkColumn (base null; only cpu_utilization declares it, as sample_time_utc) and CollectorContext gains WatermarkFromUtcColumn. Both hosts branch on the declaration: a definition without a twin goes through the unchanged GetLastCollectedTimeAsync and its byte-identical SQL; a definition with one goes through GetLastCollectedTimeWithFrameAsync, ONE statement reading MAX(twin), MAX(local), handing back the twin in the UTC frame where any row carries it and the local maximum in the LOCAL frame otherwise. The collector compares each row in the watermark's frame. The first post-upgrade run finds no twin and dedups local-to-local exactly as before (zero duplicated, zero dropped beyond the old rule); every run after it is UTC-to-UTC. Not the gate, not the de-skew readers, not sample_time's frame, not Azure (server-side filter on end_time, UTC in both columns; query text and parameter pinned identical in either frame). No rung. Pins: the three acceptance sequences through the real ReadAsync (fall-back hour lands under UTC and the local control lands zero; the upgrade day at zero duplicates; Azure identical), the never-path kept-not-guessed, the census (only cpu_utilization declares the twin, none beside a PerDatabaseWatermarkColumn); both hosts' SQL as shipped strings and their branches as source; the DuckDB pair read on a real store; the PostgreSQL round trip reading the runner's pair read at three moments; the #2797 IL gate pin tracking both readers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Partial #3653 — item 13 (Q7, Q8)
Darling V134 / Lite v63 — the "time honesty" rung. Two nullable columns that let the store say, for the first time, which clock two of its values are in:
cpu_utilization_stats.sample_time_utc(the same instantsample_timerecords, in UTC) andserver_properties.time_zone_id(the engine's own zone name, beside theutc_offset_minutesV16 added). Both SKUs, one PR, per the ruling's sequencing note.The mechanism — two lies about time
cpu_utilization_stats.sample_timeis the monitored server's LOCAL wall clock in a store whose every other timestamp is naive UTC. That is a documented convention, not an accident (CollectorTimestampFrameTestspins it): on the ring-buffer arm it isSYSDATETIME()minus each entry's age, Lite plots it in the server's own frame, and it is the collector's watermark. The cost of the convention is that every reader that aligns it against UTC has to derive the UTC value. Darling's #1262 de-skew recovers the offset per batch asMAX(sample_time) OVER (batch) − collection_timerounded to fifteen minutes; Lite shifts its whole window by the oneutc_offset_minutesthe store holds now. Both assume one offset per batch or per server, and across a DST transition that is false for exactly the rows nearest the change. Because the clock falls back, an older EDT-side sample carries a LATER local stamp than the newer EST-side one in the same poll, so the batch's MAX recovers −4 h and places the EST-side samples an hour early — silently, in the plausible direction. Lite's window applies today's offset to yesterday's samples and misses an hour's worth on the far side (the live test seeds exactly that shape and the pre-rung predicate returns[11]where the fixed one returns[22, 11]).server_properties.utc_offset_minutesis an OFFSET —DATEDIFF(MINUTE, GETUTCDATE(), GETDATE()), the one in force at collection. #3231 already documented that subtracting it from a stored instant that can outlive a transition (get_index_usage'slast_user_access, the PVS cleaner stamps) is exact on one side and an hour wrong on the other, and that nothing in the store can say which side a given instant was on. An offset cannot; a zone can, becauseAT TIME ZONEapplies the rule in force at the instant.What the rung does
CpuUtilizationCollector): every arm projectssample_time_utcLAST. Ring buffer: the identical two-stepDATEADD(the [BUG] Lite - CPU Graph renders dots and no lines #2749 millisecond precision and the MemoryPressureEventsCollector: arithmetic overflow after #2751's millisecond-precision fix #2755 overflow split both apply) anchored onSYSUTCDATETIME()instead ofSYSDATETIME()— both are runtime constants folded once per statement, so the pair differs by exactly the server's offset. Azure SQL DB:drs.end_time, whichsample_timealready reads;end_timeis documented UTC and Azure's clock IS UTC, so the two frames are one value there and neither column lies.sample_timeis not converted and the watermark stays on it: existing rows would be indistinguishable from converted ones, and a watermark that changed frame would re-ingest or skip one offset's worth of samples on the first poll after the upgrade.ServerPropertiesCollector):time_zone_id = CURRENT_TIMEZONE_ID(), read in its ownsp_executesqlbatch behindProductMajorVersion >= 16 OR EngineEdition IN (5, 8)and insideTRY … CATCH. This is the one column in the file that genuinely needs a version gate and the comment says so rather than dressing it as the deferred-object-ref pattern: on SQL Server 2019 and earlier the function is a missing BUILT-IN, not a missing object, so a batch that names it fails to compile as a whole —OBJECT_ID()has nothing to test and aCASEcannot help. The dynamic batch compiles one level down, so a compile error there IS catchable. NULL where the engine cannot say; the main SELECT still reads no table and no function that could fail to compile (MainQuery_ReadsNoTable_…stays green).Migration(134, "time-honesty")— twoADD COLUMN IF NOT EXISTS, nullable, no default, no backfill;v_cpu_utilization_statsrefreshed for the V14 reason (server_propertieshas no view). Lite v63: the same two ALTERs, non-fatal per statement, v_ views rebuilt on start. Generated CREATE TABLE carries both columns last on a fresh store. Viewer probe gains the V134 sentinel (cpu_utilization_stats.sample_time_utc, ordinal 109) and the top arm; V133's tests hand over the top-rung claims.sample_timecould be converted only by the offset its server had AT THAT SAMPLE'S INSTANT, and the store never recorded that — it holds the sparse on-load offset series and the per-batch estimate, which are exactly the DST-blind derivations this column replaces. A backfilled value would be the old lie written into the new column where a reader could no longer tell it from a measurement. And no zone can be recovered from an offset: several zones share every one.get_server_properties(both SKUs, byte-identical keys):utc_offset_minutes,time_zone_id, andtime_zone_note— "time_zone_id is null: a pre-2022 engine (…), so only the offset in force at captured_at is known." The tool descriptions say the same.Reader table — every
cpu_utilization_statsreader on dev, and what it does nowViewerDataService.Cpu.CpuUtilizationSql(Darling viewer CPU tab + Overview lane)COALESCE(sample_time_utc, <#1262 de-skew>) AS sample_timeForDisplay; post-rung rows placed by the stored instant, pre-rung rows exactly as before; window stays oncollection_timeDarlingDataReader.CpuUtilizationSql(MCPget_cpu_utilization)hours_backwindow; the payload field keeps its name, its value is UTC either wayLocalDataService.Cpu.GetCpuUtilizationAsync(Lite chart + MCPget_cpu_utilization)COALESCE(sample_time_utc, sample_time − to_minutes($offset)) BETWEEN utcStart AND utcEndsample_time − offset ≥ utcStartis algebraically the oldsample_time ≥ utcStart + offset, so the pre-rung population selects byte-for-byte as before. The projectedSampleTimestays server-local because the chart plots the server's frame and the local stamp IS that frame with no offset appliedDarlingWorker.LatestCpuSql(CPU alert read)sample_timehere is the persistence gate's observation identity, compared by>against the value the same read stored last sweep. Switching it to the twin changes the frame exactly once, at the upgrade, so every server EAST of UTC would see its first post-rung sample read older than its last pre-rung one and the gate would freeze for one offset's worth of hours (#3282's trap). Doc says so; pinnedDarlingHealthReader.ServerSummaryCpuSql,DarlingFleetReader.FleetCpuSql,ViewerDataService.Overview.ServerSummaryCpuSql, LiteLocalDataService.OverviewlatestORDER BY collection_time DESC, sample_time DESCcarries no clock frame; within one batch both columns order identically outside a DST-straddling pollPgAnomalyDetector.CpuWindowSql/HasBaselineDataSql,PgBaselineProviderCPU,PgDrillDownCollector.SpikePeakSql,PgFactCollector.Resources.CpuUtilizationSql,DailySummarySqlcpu, FinOps ×3 (Darling);AnomalyDetector,BaselineProvider,DrillDownCollector,DuckDbFactCollector.Resources,DailySummary, FinOps ×4 (Lite);DarlingDataReader.CpuWindowAggregateSql/ LiteGetCpuWindowAggregateAsynccollection_time, the collector prefix, which was always naive UTC; none readssample_timeat all. The brief's "PgAnomalyDetector.cs:151" and "alert engine High CPU window" turned out to be this shape (claim found not applicable — see the report)MeasureCatalogCompose CPU measuresPrefixTimeColumnName=collection_timeand never readsample_time;ComposeCompiler'smake_interval(mins => utc_offset_minutes)shift is for annotation TimeColumns, not this table. If a sample-grain CPU measure keyed onsample_timeis ever added, it should useCOALESCE(sample_time_utc, …)RdsCpuIngestor,DarlingMcpPgCpuUtilizationTools,FleetPgCpuSql,memory_pressure_events/memory_grant_statsreaderspg_cpu_utilization.sample_timeis UTC from Performance Insights;memory_pressure_events.sample_timeis UTC per #2932What it does NOT do
It does not convert
sample_time, rename it, or move the watermark. It does not backfill either column. It does not applyAT TIME ZONEanywhere yet — the zone is stored so a later lane can fix the #3231 de-skews at their read boundaries, and until then the offset does what it did, which is why the offset is kept rather than replaced (it is NULL-zone on most of today's fleet anyway). It does not touchMeasureCatalog(fenced; nothing needed),PgAnomalyDetector's window SQL (L1's, and it never readsample_time), the analysis windows, or anyPgTarget*file. It adds no viewer surface for the zone:get_server_propertiesis the surface; the viewer's server tab has no properties grid without XAML surgery, so that is skipped and said here.Evidence
TZ=America/New_York(mine, torn down): the ring-buffer batch returnssample_time = 07:49:29.9864135,sample_time_utc = 11:49:29.9864135—DISTINCTdifference over every ring-buffer row is exactly14400000ms;server_propertiesreturnsutc_offset_minutes = -240,time_zone_id = 'Eastern Standard Time'. On a UTC container the difference is 0 ms and the zoneUTC. Both Azure arms compile with the fourth column in place.DARLING_TEST_PG, mine, torn down): a store stopped at 133 (columns dropped, view narrowed) climbs to 134 in exactly one rung, then 0; both columns nullable, default-less, LAST; the refreshed view serves the twin; two rows through the realCpuUtilizationCollector.WritePayloadover binary COPY in a fall-back-straddling batch plus two planted pre-rung rows — MCP and viewer reads return[pre1, pre2, straddle, newest]at their true UTC instants, and a control shows the de-skew alone puts the EST-side sample atnewestUtc − 1 h;LatestCpuSqlreturns the local stamp;ServerPropertiesCollector.WritePayloadwrites-300 / 'Eastern Standard Time'and-240 / NULL, andget_server_propertiespublishes each with the right note.v_views re-created OVER the narrower tables) climbs to 63 with the views present; both columns come back nullable at the tail; pre-rung rows read NULL through the re-expanded views; the Lite CPU window selects the post-DST row the offset-shifted window missed (red-first: old predicate[11], new[22, 11]).Tests
New:
Darling.Tests/TimeHonestyRungTests(rung SQL/doc pins, generated-last + COPY column, top-arm probe, both windowed reads' projection byte-identical, latest-row reads keep local + worker doc, properties read/payload, Lite twin pin; live class as above) andLite.Tests/TimeHonestyRungTests(v63 block pins, generator trailing columns, MCP description parity, CPU window DST test, picker-range test,get_server_propertiespayload, v62→v63 upgrade). Extended:CpuUtilizationCollectorDefinitionTests(+4: twin arithmetic offSYSUTCDATETIME()and only there, Azure arms projectend_timelast, writer emits 4 values null-or-value),ServerPropertiesCollectorDefinitionTests(+2: gated dynamic batch shape, NULL zone stays NULL; 23 columns),CollectorTimestampFrameTests(+1: local clock feedssample_timeonly, UTC clock feedssample_time_utconly, identical arithmetic),ConsumedTimestampFrameDisciplineTests(census 67→68 / 49→50 / Utc 12→13,cpu_utilization_stats.sample_time_utc=Utcin the classified sequence),GoldenCollectorSchema(two appends),PgNumbackendsAndSampledMsRungTests(top-rung handoff; its live climb now stated against the ladder rather than as 1),PerfmonCounterTypeTests(v61 climb pinned toCurrentSchemaVersion),AgedDatabaseMigrationTests(named columns in the v47 fixture's INSERT — the first rung since v47 to widenserver_properties— plus an assertion the v63 column landed).Executed: whole
Darling.Testson this Mac with the rig — 11,157 total, 191 platform reds (WPF/SqlClient/Windows-path; dev baseline 188–192), zero lane-related; Lite harness 305 total, 301 green, the 4 reds the knownIgnoredWaitTypes.Load → PresentationFrameworkartefact in wait-stats MCP tests untouched here. Every touched project builds at 0 warnings.