Normalize query_stats/procedure_stats payloads into hash-keyed dimension tables (#1767)

[erikdarlingdata]

Closes #1767.

query_text / query_plan_xml stored inline on every collected row were 94% of a 250 GB production field store. This normalizes both into hash-keyed dimension tables, zero-rewrite.

Why the key is a content digest, not query_hash / query_plan_hash

The issue sketched query_text_dim(query_hash PK, ...). I did not use that, and the departure is the main design decision to review:

Candidate key	Why it fails
query_plan_hash	A plan-SHAPE hash. The same shape recompiled with different sniffed parameters carries the SAME hash and DIFFERENT XML, so ON CONFLICT DO NOTHING pins the first XML forever and serves stale ParameterCompiledValues -- the most diagnostic part of a plan for this product's audience.
query_hash	A query-SHAPE hash. Literal variants collapse, pinning one arbitrary literal set forever, where every reader today shows the LATEST text.
plan_handle	Does not determine the XML at all. query_stats captures the STATEMENT plan via dm_exec_text_query_plan(plan_handle, statement_start_offset, statement_end_offset) and the offsets are not stored. This codebase already recorded that exact failure -- QueryStatsCollector's delta key: "keying on plan_handle alone cross-contaminated multi-statement plans (one plan_handle, many statements at different offsets)".

A SHA-256 content digest is correct by construction and rests on no SQL Server cache-internals assumption. It costs two bytea columns per query_stats row -- ~1% of the payload it removes -- and it does not cost dedup ratio, because the repetition being collapsed is the same cached plan re-read each cycle.

Measured on a live SQL Server to confirm exactly that: two collections 20 seconds apart, 69 of 89 cached plans hashed byte-identical. The other 20 were plans that entered or recompiled between snapshots on an otherwise idle instance where the measurement itself perturbs the cache. That byte-identity is the property the whole design turns on, so it was verified rather than assumed.

Zero-rewrite migration

Per the trap rejected in #1759, there is no backfill and no column drop. V38 adds the two dimension tables and three nullable digest columns and touches not one existing row (metadata-only). Inline columns REMAIN, new rows leave them NULL, every reader coalesces, and raw retention ages the inline copies out on its own. The hourly/daily CAGGs never carried the payload, so nothing above raw is affected. Dropping the columns is a later, separate decision, deliberately not taken here.

StorageVersion 37 to 38, with the viewer probe sentinel (query_plan_dim) and map arm. Service deploys before viewer, per house practice -- and here it genuinely matters: a pre-V38 viewer against a V38 store reads NULL text and plan for every new row, with no error.

Consumer inventory (the contract)

Grepped the whole repo per-citation. 21 in-scope citations; everything else dismissed with a reason (other tables carrying same-named columns -- deadlocks, blocked_process_reports, query_snapshots, query_store_stats; the deprecated Dashboard; Lite; install/).

Mechanism: v_query_stats is rebuilt as a payload-RESOLVING view (COALESCE(inline, dim)) whose column list is generated from the collector definition, so it cannot go stale the way a hand-written list would -- the exact defect V14 exists to fix, not re-introduced.

Consumer	Disposition
9 query_stats readers (viewer plans, top queries, heatmap, comparison, FinOps high-impact, MCP stored-plan + data reader, actual-plan resolver, 2 analysis plan advisories)	read v_query_stats
6 readers already on v_query_stats (bad-actor, top-CPU, spilling, parameter-sensitive, bad-actor detail, expensive queries)	verified, no edit -- fixed free by the view
2 procedure_stats plan readers	explicit LEFT JOIN query_plan_dim, IS NOT NULL guard moved onto the COALESCED expression
2 presence flags (has_query_plan)	OR ..._digest IS NOT NULL -- no join needed to test presence
Top Queries aggregate CTE	deliberately still the BASE table: it needs only a presence flag over the whole window, and the view would make Postgres join the plan dimension per row to evaluate it
compose/MeasureCatalog, Custom Views, viewer wwwroot JS	not in scope, verified: no measure or dimension is a payload column, and the web UI renders no plans at all
RetentionTierRouter.RawTextHorizon	value unchanged; doc extended (CAGGs still carry no payload, and dim GC is pinned to the FACT horizon)

The residual risk is silence -- a future reader selecting the raw column does not error, it returns NULL for every new row, which looks like a collection outage. So there is now a source-parsing guard over every Darling .cs: any SQL literal reading a payload column off a bare fact table must also read v_query_stats, name a digest column, or COALESCE the payload. It passes every current site with no allowlist.

Dimension GC -- decided with numbers

Orphaned dim rows must be bounded: ~23 MB/hour of distinct plans measured in the field is ~552 MB/day, ~16.6 GB/month, ~200 GB/year if kept forever -- which would re-create the problem this PR removes. The obvious sweep (delete dims no live fact references) is an anti-join against two hypertables per dim row and is not affordable at this size.

Implemented as a last_seen watermark: the write path stamps it on every cycle that references a digest, so it is never older than the newest fact referencing it, and GC becomes an index range scan riding the existing retention sweep. The horizon is the widest effective FACT retention, resolved through the same resolver the fact purge uses -- so a raised per-collector override can never outlive the dims and orphan a reader -- plus a margin for drop_chunks granularity (a chunk survives until its whole range is past the cutoff) and the watermark's own hourly refresh guard. That guard caps the update churn at one per row per hour instead of one per row per cycle.

Autovacuum (issue item 3) -- no gap, demonstrated

The field observation (pg_stats 0 rows for these columns) is the expected, benign case: PostgreSQL never builds inheritance-parent statistics when only children are modified, and a hypertable parent holds no rows of its own. Reproduced on the rig:

PARENT pg_stats rows | 0      <- matches the field observation exactly
CHUNK  pg_stats rows | 56     <- the statistics exist, on the chunks
chunks autoanalyzed  | 2 of 3

The planner uses chunk-level stats, which exist and are being maintained. No remediation needed and none added. Darling writes no setting that suppresses autoanalyze (verified: zero hits for autovacuum = off, track_counts, autovacuum_analyze_*, default_statistics_target); its only table-level knobs are insert-triggered VACUUM thresholds, which are more aggressive than default and do not touch analyze.

Lite parity -- investigated, correctly no-op

Lite has the same inline schema (shared collector definitions) but has never stored a single plan: CollectorContext.CapturePlanXml defaults false and Lite never sets it -- confirmed empirically, COUNT(query_plan_xml) = 0 across 15,490 live rows and a 1,191,596-row monthly parquet archive. Its query_text measures 1.72 MB compressed per server-month (345x on that column, 5.8% of the file) against a hot store hard-capped at 512 MB by an archive-and-reset trigger. There is no problem to port a mechanism to. Documented rather than mechanized. Lite.Tests: 1580 passed, 0 failed.

Cheap levers -- all three dispositions contradict the issue's premise

• Compression delay 24h to 6-12h: already at the floor. CompressAfterDays is 1 (not 24h-and-tunable) and the binding constraint is ChunkIntervalDays = 1 -- a chunk cannot compress before it CLOSES, so lowering the delay alone is a no-op. Getting the intended effect means a sub-day chunk interval, which is a different change with planning-time and drop_chunks-granularity consequences, and after this PR the hot window is ~13 GB/day instead of ~89 GB/day, so it is chasing a problem the dedup removes. No change.
• Raw retention below 4 days: proposed and declined, with reason. 4 = 3 (the hourly CAGG's own refresh window) + 1 safety day, so the raw drop never outruns the aggregate preserving it. Going lower requires shortening the refresh window first -- it is structurally pinned, not a free knob. No change.
• TOP (200) provenance: deliberate in direction, inherited in magnitude. Raised 50 to 200 in 2674456a (2026-02-23) to widen coverage after a recency filter freed up slots. No rationale for 200 specifically, no storage analysis, and it predates plan capture by 4.5 months. procedure_stats' TOP (150) is pure v1.0.0 inheritance with no rationale at any point. Proposed, not changed -- row caps do not bound bytes (the whole lesson here), and after dedup a marginal row costs numerics only.

Tests -- 3,579 Darling + 1,580 Lite, and the mutation table

Every guard verified RED by build-and-run against a real PostgreSQL 18.4 + TimescaleDB store. A green local run is no evidence for which relations a query reads -- those failures are swallowed, not thrown.

Counts corrected after review. Mutation 4 was first reported as 3 tests; re-run properly it is 1, identical under a fast-only run and a gated run. The original 3 came from a filtered run against a store still dirty from the mutation-2 experiment, where two unrelated live tests were failing for that reason. The reviewer caught the discrepancy and was right. Every other row below was re-confirmed.

#	Mutation	Result
1	Remove ON CONFLICT from the dim upsert	RED -- 23505: duplicate key value violates unique constraint "query_text_dim_pkey"
2	Drop the COALESCE from the resolving view (fresh store, dim-only row)	RED -- System.InvalidCastException : Column 'query_text' is null (the exact silent-NULL symptom)
3	Drop one payload-index increment in the COPY writer	RED -- Collector wrote 10 payload values but declares 50 payload columns
4	Re-add v_query_stats to AllPassthroughViews	RED -- 1 test, DarlingObservabilityTests.MigrationScripts_AreRegisteredInAscendingOrder_...: Assert.DoesNotContain() Failure: Item found in collection / Found: "v_query_stats"
5	Commit the fact COPY before flushing dimensions	RED on the ordering pin
6	Revert one procedure_stats reader to the pre-#1767 form	RED -- source guard names ViewerDataService.Plans.cs:116 -> "FROM procedure_stats"

Mutation 5 is the one worth reading. The live atomic-visibility test MIRRORS the write path rather than calling it (WriteBatchAsync is private and only reachable through a live SQL Server), so reordering production left it green. The seam between the mirror and the real thing needed its own pin. A test I did not watch fail is not evidence, and that one proved it.

31 new tests (20 fast + 7 gated live + the visibility and ordering pins). The fast pins derive expected ordinals FROM the collector definition rather than hardcoding them, so they track a definition change instead of rotting.

Design decisions worth a reviewer's attention:

• Why not literal dims-before-facts ordering? The digests are only known once the rows stream, and a second WritePayload pass is not available -- WritePayload CONSUMES delta state (context.Deltas.CalculateDelta), so a second pass would report every delta as zero. One transaction gives the stronger guarantee (no session observes a fact whose digest resolves to nothing) and is what the test asserts.
• Why EndPayload on every collector? A payload-count drift silently shifts every later column by one, and with a diversion plan active it would hash the WRONG column. Without the check the symptom is an opaque Npgsql type error several columns downstream of the mistake.

Not predicted by the design, flagged not absorbed

1. v_query_stats was still in AllPassthroughViews -- the list GenerateV14RefreshViews() re-emits as SELECT *. "ADD COLUMN, then refresh the views" is the established idiom (V14, V15, V27, V28); applied to this view it would silently overwrite the COALESCE. Fixed structurally (excluded, with the cross-check widened to the union) plus a tripwire that the LAST migration defining the view is the resolving one.
2. DarlingDataReader and DarlingTrendReader carried a factually wrong doc claim -- "the three query-perf tables have NO v_* view". Both v_query_stats and v_query_store_stats do exist. Uncorrected, that is precisely the belief that makes the next reader select the raw column and get silent NULLs. Corrected.
3. A live-test flake, pre-existing. Across ~8 full runs, 3 single-test failures in 2 different live classes (ExcludedDatabasesStoreLiveTests, ViewerBlockingStatsLivePostgresTests), each green in isolation (3/3) and on re-run. Neither class touches query_stats; both use their own fixed server ids and read other tables. Consistent with the known shared-rig live-PG flake class, not introduced here -- recorded so it is not mistaken for this change later.

Verification

• Rebuild -t:Rebuild, 0 Warning(s) 0 Error(s) on all 7 affected projects: Darling.Storage, Darling.Analysis, Darling.Service, Darling.Viewer, Darling.Tests, PerformanceMonitorLite, Lite.Tests.
• Darling.Tests 3,579 passed / 0 failed / 7 skipped against live PostgreSQL 18.4 + TimescaleDB 2.28.1; Lite.Tests 1,580 passed / 0 failed.

Do not enable auto-merge -- a release cut is pending from the current validated lineage and this must not enter dev before it ships.

Generated with Claude Code

---

FIELD VERIFICATION RECIPE

Operator-runnable proof that the dedup is working on a real store. Every query below was executed against a live PostgreSQL 18.4 + TimescaleDB 2.28.1 store seeded with both pre-upgrade (inline) and post-upgrade (hash-only) rows — the outputs shown are real, not illustrative. All checks are READ-ONLY.

Substitute <server_id> and set <upgrade_time> to when the service was restarted on the new build (from the Postgres store ready (schema v38...) log line in E1).

PRECONDITION — upgrade the viewer in the SAME step as the service

This is the one way to get a bad outcome from a correct migration. The viewer's connect gate is one-directional: it refuses a store older than it requires, and cannot refuse one that is newer. A pre-#1767 viewer reads a V38 store as 37, 37 < 37 is false, so it connects and works — but every text and plan read in that binary hits the bare column and gets NULL.

Operator symptom of the skew, in the order it presents: the Query Text column is blank across Top Queries; the "Query Plan" download button is greyed out on every row; and "View Plan" raises a dialog titled No Plan Available reading, verbatim:

No execution plan was captured for this row. The plan may not have been collected yet, or it aged out of the store's retention window.

That presentation is indistinguishable from a collection outage — the message itself offers "not collected yet" and "aged out" as the explanations, and neither is what actually happened. So if this is observed, verify the viewer build BEFORE investigating collection, retention, or the store. Checks A and B below settle it in seconds: if the dimensions are populating and no digest is unresolvable, the data is present and the viewer is the problem.

A — the dimensions exist and are populating

SELECT d.dim, d.rows, pg_size_pretty(pg_total_relation_size(d.dim::regclass)) AS on_disk, d.newest_seen FROM (SELECT 'collect.query_text_dim' AS dim, count(*) AS rows, max(last_seen) AS newest_seen FROM collect.query_text_dim UNION ALL SELECT 'collect.query_plan_dim', count(*), max(last_seen) FROM collect.query_plan_dim) AS d;

          dim           | rows | on_disk |        newest_seen
------------------------+------+---------+----------------------------
 collect.query_text_dim |   40 | 48 kB   | 2026-07-27 16:06:00.097867
 collect.query_plan_dim |   12 | 48 kB   | 2026-07-27 16:06:00.099198

PASS: both tables exist, rows > 0, and newest_seen is within the last few minutes. Run it twice a few minutes apart — rows grows quickly at first, then flattens as the distinct-content set saturates (flattening is the mechanism working, not stalling).

FAIL — stop the run: the tables do not exist (migration did not apply — check E1); or rows is 0 after two or more collection cycles; or newest_seen does not advance between samples. Any of those means payloads are not being diverted.

B — new fact rows are hash-only, and every hash resolves

SELECT count(*) AS rows_since_upgrade, count(*) FILTER (WHERE query_text IS NOT NULL) AS inline_text_still_written, count(*) FILTER (WHERE query_plan_xml IS NOT NULL) AS inline_plan_still_written, count(*) FILTER (WHERE query_text_digest IS NOT NULL) AS carrying_text_digest, count(*) FILTER (WHERE query_plan_digest IS NOT NULL) AS carrying_plan_digest FROM collect.query_stats WHERE collection_time >= '<upgrade_time>'::timestamp;

 rows_since_upgrade | inline_text_still_written | inline_plan_still_written | carrying_text_digest | carrying_plan_digest
--------------------+---------------------------+---------------------------+----------------------+----------------------
               2000 |                         0 |                         0 |                 2000 |                 2000

PASS: both inline_*_still_written are 0. The carrying_*_digest counts are normally close to rows_since_upgrade but need not equal it — a row whose DMV returned no text or no plan legitimately carries a NULL digest.

FAIL: any non-zero inline_*_still_written means the writer is not diverting — the old inline path is still live. Do not proceed; the store is still growing at the old rate.

Then the resolution check, which must be zero:

SELECT (SELECT count(*) FROM collect.query_stats f WHERE f.query_text_digest IS NOT NULL AND NOT EXISTS (SELECT 1 FROM collect.query_text_dim d WHERE d.digest = f.query_text_digest)) AS unresolvable_text, (SELECT count(*) FROM collect.query_stats f WHERE f.query_plan_digest IS NOT NULL AND NOT EXISTS (SELECT 1 FROM collect.query_plan_dim d WHERE d.digest = f.query_plan_digest)) AS unresolvable_plan;

 unresolvable_text | unresolvable_plan
-------------------+-------------------
                 0 |                 0

PASS: both 0 — every digest on a fact row has its content.

FAIL — stop the run and report it: anything above 0 means fact rows were published whose payload is unrecoverable. That should be impossible (the COPY and the dimension upsert share one transaction), so a non-zero here is a genuine defect, not a tuning issue. Note the dimension GC is pinned to the fact horizon plus a margin, so ageing cannot produce this either.

C — the read path resolves BOTH directions

The single most important check: the transition window has to serve old inline rows and new dimension rows through the same read.

SELECT 'pre-upgrade (inline)' AS row_kind, collection_time, left(query_text, 44) AS resolved_text, query_plan_xml IS NOT NULL AS plan_resolved FROM collect.v_query_stats WHERE server_id = <server_id> AND query_text_digest IS NULL ORDER BY collection_time DESC LIMIT 1;
SELECT 'post-upgrade (dimension)' AS row_kind, collection_time, left(query_text, 44) AS resolved_text, query_plan_xml IS NOT NULL AS plan_resolved FROM collect.v_query_stats WHERE server_id = <server_id> AND query_text_digest IS NOT NULL ORDER BY collection_time DESC LIMIT 1;

       row_kind       |      collection_time       |               resolved_text                | plan_resolved
----------------------+----------------------------+--------------------------------------------+---------------
 pre-upgrade (inline) | 2026-07-24 16:39:20.081094 | SELECT * FROM dbo.Orders WHERE OrderId = 0 | t

         row_kind         |      collection_time       |               resolved_text                | plan_resolved
--------------------------+----------------------------+--------------------------------------------+---------------
 post-upgrade (dimension) | 2026-07-27 15:16:02.099873 | SELECT * FROM dbo.Orders WHERE OrderId = 0 | t

PASS: BOTH queries return a row with non-empty resolved_text and plan_resolved = t.

FAIL: the post-upgrade query returns NULL text or plan_resolved = f → the view is not resolving the dimension. If the pre-upgrade query returns no row at all that is not a failure — it just means raw retention has already aged every inline row out, which is the expected end state a few days after the upgrade.

Then confirm it through the product, not only SQL (this is what the acceptance test is really for): in the viewer, open Top Queries, pick a row collected after the upgrade, and use View Plan and Download .sqlplan. Both must open the plan. Repeat on Top Procedures (its plans resolve through a different code path — an explicit dimension join, not the view). Then FinOps → Expensive Queries and High Impact Queries, which carry plan XML in-row rather than fetching on click. Blank text or a greyed-out plan button on a post-upgrade row is a FAIL — and re-read the PRECONDITION above before concluding anything, because viewer skew produces exactly that symptom.

D — the growth-rate proof

Inline payload bytes written per hour. The pre-upgrade hours carry the old cost; post-upgrade hours must go to zero.

SELECT date_trunc('hour', collection_time) AS hour, count(*) AS rows, pg_size_pretty(COALESCE(sum(pg_column_size(query_text) + pg_column_size(query_plan_xml)), 0)::bigint) AS inline_payload_bytes FROM collect.query_stats WHERE server_id = <server_id> GROUP BY 1 ORDER BY 1;

        hour         | rows | inline_payload_bytes
---------------------+------+----------------------
 2026-07-24 16:00:00 | 2000 | 168 kB
 2026-07-27 15:00:00 | 2000 | 0 bytes

PASS: hours after the upgrade read 0 bytes while row counts stay in the same range as before. On this instance the baseline was ~3,166 MB/hour; the post-upgrade figure for a comparable hour should be 0 for the fact table, with the real cost now being only the new distinct content added to the dimensions (D3).

FAIL: a post-upgrade hour with non-zero inline bytes (same defect as B), or a collapse in rows (that is a collection problem, not a dedup success — the dedup must not change how many rows are collected).

Chunk-level, the same instrument used for the original finding:

SELECT c.chunk_name, c.range_start, c.range_end, c.is_compressed, pg_size_pretty(s.total_bytes) AS total, pg_size_pretty(s.toast_bytes) AS toast FROM timescaledb_information.chunks AS c JOIN chunks_detailed_size('collect.query_stats') AS s ON s.chunk_name = c.chunk_name AND s.chunk_schema = c.chunk_schema WHERE c.hypertable_name = 'query_stats' ORDER BY c.range_start;

PASS: chunks whose range_start is after the upgrade show toast collapsing toward nothing — that is where the payload lived (the original finding measured 74.15 GB TOAST of an 88.97 GB chunk, and 16.73 GB of the 17.98 GB compressed chunk). Compare a fully post-upgrade chunk against a fully pre-upgrade one; note ranges are UTC-aligned, so a chunk labelled with one date may cover part of the next.

Interpretation note: on a store with small query text and no plan capture, toast reads near zero before the change too, because values under ~2 KB stay in the heap. On this instance plan XML averages several KB and is genuinely TOASTed, which is why TOAST is the right column here.

Dimension footprint — sample twice, an hour apart; the delta is the real hourly cost that replaced the ~3,166 MB:

SELECT pg_size_pretty(pg_total_relation_size('collect.query_text_dim') + pg_total_relation_size('collect.query_plan_dim')) AS dimensions_total;

PASS: the hourly delta is small and shrinking as the distinct-content set saturates. FAIL: growth tracking row count linearly (each row producing new distinct content) would mean the digest is not collapsing repeats — check that the same plan re-collected yields one dimension row via check A.

E — log lines to pin

E1 — migration applied, once, on the first start of the new build (DarlingWorker):

Postgres store ready (schema v38, 1 migration(s) applied)

A later restart logs schema v38, 0 migration(s) applied. FAIL: a version below 38, or the line absent — the service did not migrate, and no other check can pass.

E2 — collection continuing at normal volume (DarlingCollectorRunner / DarlingWorker, one line per collector per sweep):

  [<server>] query_stats => 200 rows (sql:123ms, pg:45ms)
  [<server>] procedure_stats => 150 rows (sql:88ms, pg:31ms)

PASS: row counts unchanged from before the upgrade (the caps are still 200 / 150 — this change deliberately did not alter collection volume) and pg: timings in the same range. FAIL: => 0 rows on either collector, or pg: timings materially worse — the dimension upsert adds two statements per batch and should be a small fraction of the COPY it replaces.

E3 — retention sweep, including the dimension GC (DarlingRetention, daily):

Retention purge: <N> table(s) purged, <R> row(s) deleted, <C> chunk(s) dropped, 0 failed, <ms>ms

PASS: 0 failed. The dimension tables are included in the table(s) purged count. FAIL: a non-zero failed count, or a Retention purge failed for query_text_dim / query_plan_dim warning.

What a fully successful run looks like

A: both dimensions present and growing, then flattening. B: zero inline writes, zero unresolvable digests. C: both directions resolve in SQL and plans open in the viewer on post-upgrade rows. D: post-upgrade hours at 0 inline bytes, TOAST collapsing on post-upgrade chunks, dimension growth small and decelerating. E: schema v38, unchanged collection volume, clean retention sweep.

Any FAIL above should stop the run and be reported rather than narrated past — every one of them distinguishes "the dedup is working" from "the payload is silently missing", and those two look identical from the outside.

[erikdarlingdata]

Field-scale correction to the FIELD VERIFICATION RECIPE in this PR body, from the first production acceptance run: the D2b unresolvable-digest query has no time bound on the fact side, so TimescaleDB cannot chunk-exclude and it scans the entire hypertable - it does not complete on a 276+ GB store (even a bare count(*) WHERE query_text_digest IS NOT NULL times out there). The corrected form adds the same collection_time >= <upgrade_time> bound D2a uses to BOTH subqueries; validated live on that store at ~32s with a clean 0/0 result. The operator runbook has been updated. Lesson for future recipes: a query executed against a seeded dev store proves correctness, not scalability - production-runbook queries need the time bound that makes chunk exclusion possible.