The Fleet page and the fleet overview total PostgreSQL deadlocks without ordering databases whose counter never moved (#5526)

[erikdarlingdata]

Fixes #5526

What changed and why

The Fleet page totals and the get_fleet_overview tool count PostgreSQL deadlocks as counter differences over collect.pg_database_stats: deadlocks - LAG(deadlocks) OVER (PARTITION BY server_id, database_name ORDER BY collection_time). That sorts and windows every row of the window. On the field store (#5508, batch 4 item 2) the first version of this PR, which differenced one server at a time, was only 2.7% faster over 30 days (26,575 ms against 27,319 ms cold), because both forms spend the time sorting and windowing all 17.1 M rows. This version stops ordering rows that cannot matter.

Most (server_id, database_name) pairs never change their counter inside a window. For such a pair every difference is 0, so ordering it buys nothing.

• One unordered aggregate over the window, grouped by (server_id, database_name): row count, count(deadlocks), min, max.
• A pair is flat when count(deadlocks) = count(*) (a counter in every row) and min = max. A flat pair of n samples has n - 1 differences, all 0. It adds 0 to cnt, nothing to last_seen and n - 1 to intervals. That is exact, not an approximation.
• Every other pair (any change, any NULL counter, a column that is NULL throughout, a counter reset) goes through the old ordered LAG, unchanged, one server at a time. The read is bounded by server_id (the hypertable's segmentby, server_id only, default order collection_time DESC) so a compressed chunk is read through its segment index, and keeps that server's non-flat pairs by name. The NULL-named shared-relation series is matched by IS NULL, because = ANY never matches NULL. A server with no non-flat pair skips the ordered read (a one-time filter in the plan).
• A server with only flat pairs still gets its row: cnt 0, last_seen NULL, intervals = the sum of n - 1. A server whose only row in the window is one sample still gets intervals 0. A server with no row in the window still does not appear. Output columns, types and row set equal the old read for every input.
• The Fleet page read (ViewerDataService.FleetTotalsSql) returns only the total, so it keeps the pairs with min(deadlocks) <> max(deadlocks) and needs no "counter in every row" half: a pair that never takes two values adds nothing to the sum of positive differences, NULLs included. The two reads are different SQL in different projects, so they stay two constants; their comments point at each other.
• Code only. No rung, no schema version, no new index.

Files: DarlingFleetReader.cs (FleetPgDeadlockSql), ViewerDataService.Fleet.cs (FleetTotalsSql), and tests.

Measurements

The 30-day window below is a measurement window only. No caller can ask for it: the fleet overview tool caps hours_back at 168, the web /api/fleet clamps to 1..168, and the viewer totals read has one caller that always passes 1 hour. The changelog entry and the source comments say so.

Field store (from the issue and #5508 comment 6053411189, batch 4 item 2; 50 targets, 17.1 M rows, 31 chunks, 29 decompressed): old form 26,854 ms cold and 26,571 ms warm over 30 days. The first version of this PR (per-server lateral): 26,575 ms against 27,319 ms cold, 2.7% faster. Not yet measured on the large store for this version; read-only SQL for it (old against new over 30 days, with a same-rows check) is written and ready to run there.

Rig numbers (not the large store). PostgreSQL 18.6, TimescaleDB 2.30.1, UTC, default work_mem, no parallel workers used. Built through the real migrate, then the shipped hypertable and compression settings (1-day chunks, segmentby = server_id). 5.22 M rows over 30.5 days at one row a minute, 20 servers plus 3 small ones, 123 pairs in the window, 32 chunks (30 compressed, 2 not). Two seeds: "5%" (9 of 123 pairs not flat: about 5% moving, plus a counter reset, a partially-NULL pair, an all-NULL pair, a single-row pair, pairs that appear mid-window, a retried-write twin, a pair and a server entirely before the window) and "all changed" (every pair moves). EXPLAIN (ANALYZE, BUFFERS), warm, 30-day window:

Seed	Read	Old	New
5%	fleet overview read, quiet machine	12,429 / 11,731 ms	2,136 / 1,936 ms
5%	Fleet page total arm, quiet machine	14,143 / 11,744 ms	1,753 / 1,604 ms
5%	fleet overview read, later pass	15,634 / 13,054 ms	5,698 / 4,226 ms
all changed	fleet overview read, quiet machine	12,871 / 11,620 ms (and 11,256 / 12,600 / 14,196 / 11,386)	9,197 / 9,137 ms
all changed	fleet overview read, later pass	11,648 / 12,913 / 13,139 / 9,960 / 18,941 / 10,960 ms	20,968 / 20,714 / 21,072 / 17,728 / 14,142 / 16,998 ms
all changed	Fleet page total arm, all passes	9,722 / 10,858 / 17,251 / 11,082 / 12,764 / 12,312 / 9,786 / 21,411 ms	14,251 / 11,696 / 14,023 / 10,453 / 11,786 / 13,389 / 25,099 / 18,486 ms
5%	1-day window, fleet overview read	813 / 658 ms	153 / 217 ms
5%	1-hour window, fleet overview read	16 / 14 ms	6.7 / 6.6 ms

What to read from this, honestly:

• 5% seed: about 5.5x to 7x faster on the quiet machine, and still about 3x in the later pass. This clears the 3x target.
• All-changed seed: the gate was no more than about 10% slower. Early on the quiet machine the new form was faster (9.2 s against 11.6 to 14.2 s). The laptop then became busy with other work and the later passes swing from 10 s to 21 s for both forms, with the new form's minimum above the old form's minimum (14.1 s against 10.0 s). I could not get a quiet pass back before the time limit, so I cannot show the all-changed cost as inside 10% with a clean measurement. The plan text is the same in both passes (no parallelism in either). Treat this row as unconfirmed and check it on a quiet machine or on the large store with the ask file. All-changed is not the field shape; the field shape is the 5% one.
• Where the new time goes (5% seed): about 1.2 s for the single unordered aggregate that decompresses every chunk, and about 0.6 s for the ordered read of the 8 servers that have a moving pair.
• Same rows: the old and new reads return identical rows on both seeds (sorted diff of the full result, 22 rows each), and the Fleet page arm's total equals the sum of the old cnt on both seeds (159 and 2,604).

Test plan

• FleetPgDeadlockFlatPairLiveTests (renamed from the first version's file): runs the previous query text (kept as a literal) and the shipped FleetPgDeadlockSql and FleetTotalsSql on one seed, whole result row by row, plus hand-worked numbers for servers with only flat pairs, NULL counters (a constant with one NULL, an all-NULL pair, NULL then moving), a moving NULL-named series, the counter reset, the one-row server, the retried-write twin and a server outside the window. Then two generated seeds (about 9,000 rows each: mixed, and every pair moving) get the same comparison.
• RED-planted, each reverted: dropping count(deadlocks) = count(*) from the service read fails the live test (server with NULL counters: 6 intervals against 4); dropping the IS NULL branch fails it (the moving NULL-named series gives count 0 against 3); inverting the Fleet page HAVING fails it (total 0 against 19).
• Shape pins updated in DarlingFleetDeadlockCoverageTests and ViewerFleetRollupSqlTests (the flat rule, n - 1, one window function, per-server filter, name and NULL-name match, both window bounds twice in both reads).
• On my rig (live Postgres): *FleetPgDeadlockFlatPair*, *DarlingFleetReader*, *DarlingFleetDeadlockCoverage*, *DarlingFleetDto*, *DarlingMcpFleetTools*, *ViewerFleet*, *FleetReadsAreBounded*, *FleetCardPostgresCpu*: 194 run, 0 failed. *Census*, *Ratchet*, *Hygiene*, *Pin*, *RepoFileAdoption*, *FleetIdentifierScrub*, *RegisteredServerSettingDrift*, *TsqlConventionGuard*, *Guard*, *Scan*, *DocComment*, *CommandTimeout*: 1,985 run, 0 failed, 2 skipped. After merging origin/dev, rebuilt with 0 warnings and re-ran the fleet pins, Census, Hygiene and Ratchet: 637 run, 0 failed, 5 skipped (no live database running then).
• Round-2 review fixes: the one-time gate WHERE (cardinality(s.ordered_names) > 0 OR s.ordered_null_name) is now pinned in DarlingFleetDeadlockCoverageTests. RED-checked: with the line removed from the SQL the pin fails ("Not found"), restored, passes. Rebuilt with 0 warnings; then *DarlingFleet*, *DarlingMcpFleetTools*, *ViewerFleetRollup*, *Census*, *Ratchet*, *Hygiene*, *Pin*, *RepoFileAdoption*: 1,243 run, 0 failed, 35 skipped (live PostgreSQL classes, no database this round).
• The live comparison runs on plain tables in the test database; the compressed-chunk and segment-index behavior was checked on the rig seeds only (the numbers above), not in a test.
• Full suite not run (CI decides).
• Large store (50 PostgreSQL targets, Audit the Darling store's large tables for reads that do not fit their indexes #5508 batch 7, issuecomment-6054791357): old and new return the same rows at 168 h and 30 d (EXCEPT both ways, 0 rows). 433 of 443 pairs flat over 168 h; the ordered pass reads 2.3% of rows. 168 h: fleet overview 6.8 s -> 0.68-0.77 s, Fleet page total 6.7-6.8 s -> 0.64-0.80 s. 1 h overview 42 ms -> 15.6 ms. No spill. The new reads touch 20-25% more buffers; the time saved is the sort and LAG.

Changelog

SECTION: Fixed
ENTRY:

• The fleet overview tool and the web Fleet page total PostgreSQL deadlocks faster over multi-day windows ([The Fleet page and the fleet overview total PostgreSQL deadlocks without ordering databases whose counter never moved (#5526) #5536]). These totals sorted and differenced every server's samples. Both surfaces accept windows up to 7 days, and on a store with 50 PostgreSQL targets a 7-day total took about 6.8 seconds. They now set aside the databases whose deadlock counter had the same value in every sample in the window and difference only the rest. On that store a 7-day total now takes about 0.7 seconds. The totals are unchanged, including after a counter or statistics reset and for databases with missing counter values.
  REF:
  [The Fleet page and the fleet overview total PostgreSQL deadlocks without ordering databases whose counter never moved (#5526) #5536]: The Fleet page and the fleet overview total PostgreSQL deadlocks without ordering databases whose counter never moved (#5526) #5536