The duration-trend trio routes by retention tier and discloses its source like get_query_trend already does, so a 7-day request no longer returns 4 days labelled quiet (#3541 A2)

[erikdarlingdata]

The duration-trend trio now reads the tier that holds the window, and every sibling says what it served

get_query_duration_trend and get_procedure_duration_trend read raw query_stats / procedure_stats only. On a TimescaleDB store the raw tier of a ROLLED table is physically dropped at TimescaleSupport.RawRetentionSpan — four days — while both tools accept hours_back up to 168. So a 7-day request came back as whatever had not aged out, under an envelope echoing hours_back: 168, with no source, no effective_start, no truncated. And when nothing survived, the empty branch asserted "This server HAS collected query stats before, so this window is genuinely quiet rather than broken — widen hours_back to find the most recent samples." That sentence is false (the rows were dropped, not absent) and harmful (a wider window reaches further into what the tier does not hold). A reasoning agent reads the absence as quiet and concludes the regression it was hunting never happened.

The complete fix already existed one read over, in the same file: get_query_trend (#2353) routes by the age of the window's oldest point onto query_stats_hourly, and publishes source / effective_start / effective_hours_back / truncated / bucket / aggregate_note. This PR wires the trio through the same decision and the same vocabulary. Partial: it completes checklist item A2 of #3541 only.

Root cause narrative

Three reads over rolled tables, three different depth stories, none of them told:

• get_query_duration_trend / get_procedure_duration_trend: raw only. Silent 4-day ceiling behind a 7-day parameter.
• get_query_store_duration_trend: already routed (get_query_store_duration_trend ranks the raw Query Store slab per call and times out at any window width on a large store #2736) onto query_store_stats_corrected_hourly + raw tail — so its DEPTH was in fact fine (raw query_store_stats is dropped only once its rollups cover it, the Retention policies run their first check immediately — no external window to pause them, caused permanent data loss #1680 arming gate, so a raw-only route means raw is complete) — but its disclosure was a source OBJECT ({tier, rollup, raw_from, note, unserved_*}) while get_query_trend's source is a tier STRING. Same key, two types, across siblings that share one serializer "so a caller can chart them on one axis without learning three field names". The campaign's "seven truncation dialects" in miniature.
• get_query_trend itself: age-only routing with no availability or coverage rung. On a plain-PostgreSQL store (no continuous aggregates — a relation named in a statement is parse-time-resolved) any window past four days answered 42P01; on a store that pre-existed its rollups and was never backfilled (Rollup CAGGs serve only materialized buckets: old windows read empty and raw purges stay held #1759) it read an empty rollup while raw, whose purge the arming gate holds paused, held every row.

What the product does now

One resolver, one ladder — DarlingTrendReader.ResolveTier(start, now, hourlyAvailable, coverage): get_query_trend's age rule (ShouldUseRawTier, with #2353's one-hour margin kept — the router's one-day margin would push every 3–4-day window onto hourly for nothing), degraded to what the store HAS (#1664: no rollup → raw, per grain) and to what it has MATERIALIZED (#1759: comparative, only ever moving DOWN to raw when raw is measured to reach further back than the rollup's floor; nulls are inert). The store's shape comes from ComposeStoreAvailability.GetRollupsAsync — the composer's existing per-data-source cache (5-minute TTL), so this is not a probe per call. get_query_trend now passes its availability flag and coverage pair through the same function; its existing callers and tests keep the age-only default.

Hourly-tier twins — QueryDurationTrendHourlySql / ProcedureDurationTrendHourlySql sum elapsed_time_sum and execution_count_sum per bucket and divide by the bucket WIDTH (HourlyBucketSecondsSql, pinned equal to TimescaleSupport.HourlyBucket), not by a LAG over neighbouring points. That closes the measurement lane's A11a where the rewrite made it free: every bucket has a real denominator, so the raw idiom's fabricated first-point zero does not exist on this tier. The trade is stated in aggregate_note: an hour the collector covered only partly reads LOW, never high, and the rollup trails the clock by up to two hours (current hour never materialized; previous one lands on the next refresh).

One disclosure block, six keys, same order, every branch, both SKUs — source, effective_start, effective_hours_back, truncated, bucket, aggregate_note, written between the request echo and the points on the data path and beside status/message on the empty path. The Query Store sibling's source becomes the tier word (rollup+raw / raw); its #2736 seam detail (rollup, raw_from, unserved_before, unserved_note) moves under a routing block emitted only when the rollup route was taken. This is a deliberate move of a shipped MCP disclosure to the shared shape; nothing but the tool and its live test consumed the old keys (swept: raw_from|unserved_before|rollup\+raw across the tree). value gains its named twin elapsed_ms_per_second (same quantity, kept beside it on the execution_count / executions_per_second precedent).

The empty branch tells the truth in three states, not two. Never-sampled stays unavailable — probed on the raw table AND, on the hourly route, the rollup, because a server disabled a week ago has no raw rows left while its rollup holds weeks. Sampled-and-quiet keeps the shared "widen hours_back" sentence byte-for-byte with Lite, but only where the tier is measured (or, unmeasured, known by construction) to reach the whole window. The third state is the one this fix exists for: the tier read does NOT reach the window's start — the message names the tier, its measured floor, which part is unserved rather than quiet, that widening cannot help, and --backfill-rollups as the remedy. A raw-route window a young store never existed for says "predates the store's history" rather than blaming retention. The Query Store sibling gets the split too: a window whose head sits below the rollup's floor with a quiet covered part names both halves.

Empty-branch coverage words — the tier's measured floor stands in for the first point: inside the window it is judged by the shared 90-minute slack (DarlingTrendReader.TruncationSlack, extracted from #2353's inline literal); beyond the window's end the served span is honestly zero and the answer is truncated outright.

Lite — LocalDataService is untouched (DuckDB keeps every collected row for the collector's retention_days; nothing rolls up). McpQueryTools publishes the same six keys with Lite's truth: source always raw, bucket per-collection / per-interval, effective_start the first point the store held, truncated by the same 90-minute slack (McpQueryTools.TruncationSlack, pinned to the same value on both sides since neither test project can reference the other's assembly). Same contract, different depth behind it.

Why this shape, and what was rejected

• Reusing RetentionTierRouter directly was rejected: its raw threshold is one DAY inside the horizon, get_query_trend's is one hour, and the task is "the same decision as get_query_trend". Widening get_query_trend's margin to a day would have moved every 73–95-hour window onto hourly and nulled its per-row columns for nothing. The ladder is built from the same Storage primitives (TierCoverage.Covers / ReachesFurtherBack) so the two rules differ only in the bottom rung's margin.
• Leaving the Query Store source object alone was rejected: a source that is a string on two siblings and an object on the third, from one serializer, is the envelope inconsistency the campaign is about.
• Replacing the raw reads' LAG with sample_interval_seconds (A11a proper) was NOT done: that column is per-key delta age (0 when unknowable), not the per-collection gap; procedure_stats does not carry it; and the viewer's and Lite's twins compute LAG, so changing Darling's raw estimator alone would make the same chart disagree across surfaces. Reported below with path:line.

Blast radius

MCP payloads only. The viewer and web endpoints do not read these tools' JSON (DarlingWebEndpoints passes the string through; the viewer reads ViewerDataService). The trio's existing fields (server, hours_back, trend[].time/value/execution_count/executions_per_second) are unchanged. get_query_trend's payload is unchanged; only its routing gains the two degrade rungs, which change its answer only where the old one was wrong (42P01 → raw; empty rollup → raw). Sentences shared with Lite (McpMissMessageParityPinTests scope) are untouched.

What it does NOT do

• Does not touch the viewer's own Performance Trends tab, which has the same raw-only read (ViewerDataService.QueryTrends.cs GetQueryDurationTrendAsync / GetProcedureDurationTrendAsync) — out of this lane's boundary; reported to the coordinator.
• Does not fix DarlingWebEndpoints.cs:1959's description of get_query_duration_trend as "Query-duration percentiles over time" (it is elapsed ms/sec) — out of boundary, reported.
• Does not add a daily rung: McpHelpers.MaxHoursBack (7 days) sits far inside HourlyRetentionSpan (90 days), so no acceptable window can outrun hourly by age.
• Does not close MCP payload-contract campaign: the agent-facing API tells the truth (~30 defects + an 11-rule contract) #3541; items A3–A16 and the contract remain.

Test plan

Compile-verified here (all four projects, zero warnings); the tests are net10.0-windows and are first EXECUTED in CI. Written to be right by construction: every live scenario below was also driven through the real tool methods against a throwaway TimescaleDB 2.28.1 / PG18 container with the migrated schema and the continuous aggregates created and refreshed over the exact ranges the tests use — the payloads the tests assert are the payloads that came back.

New, ungated (DarlingMcpTrendToolsTests.cs):

• DurationTrendHourlySql_ReadsTheRollup_BucketsByIt_ProjectsTheSharedShape — both hourly twins read their TimescaleSupport view constant, bucket-window, project the raw read's three aliases, and name columns the CAGG definitions carry.
• DurationTrendHourlySql_DividesByTheBucketWidth_NotALag — no LAG( on the hourly tier; the literal divided by is TimescaleSupport.HourlyBucket.TotalSeconds; the raw reads still LAG (the A11a residual is reported, not silently rewritten).
• TruncationSlack_IsNinetyMinutes_AndDescribeCoverageAppliesIt.
• DurationTrendTierRoutingTests (new class): the CENSUS — for every accepted hours_back, both plan-cache routes land where get_query_trend's age rule lands; availability degrade per grain; the comparative coverage rule (hourly kept when raw is not measured deeper; raw taken when it is; nulls inert; raw-age windows never promoted); the route names its own pair and source word.
• The dialect theory gains the two hourly SQLs.

New, gated (DarlingPerformanceTrendsReadTests.cs, own scratch store, not in live-postgres): DarlingPerformanceTrendsTierRoutingLiveTests.PastTheRawHorizon_TheTrio_ServesTheHourlyRollup_AndSaysSo — fixed March-2026 instants so the window is past the horizon on any day it runs; exact rates off the seed (10,800 ms / 3,600 s = 3.0 ms/s, 108 / 3,600 exec/s, …) for both siblings; the disclosure block; the raw-only read of the SAME fixture as revert-proof (three per-collection points, the first rated ZERO by the LAG idiom); never-sampled on the hourly route; the unserved-head empty branch with its message and clamped coverage words; get_query_trend over the same rows agreeing on tier and effective_start.

Changed, gated: DarlingPerformanceTrendsReadTests (shared fixture) asserts the block on all three siblings' data and empty envelopes, elapsed_ms_per_second == value, effective_start = first point, truncated where the head sits past the slack. QueryStoreTrendRoutingLiveTests asserts the SAME facts as before on the moved keys (source string, routing.*), adds the coverage words, and adds a tail-only second server for the partial-head empty branch and the not-truncated tail read.

Lite (PerformanceTrendsToolTests, TrendEmptyParityToolTests): the block on data/empty/never envelopes for all three tools, elapsed_ms_per_second, effective_start = first point, truncated both ways (a series beginning within the slack is not truncated; one beginning hours late is), McpQueryTools.TruncationSlack == 90 min.

Container torn down after verification.

Which component(s) does this affect?

• Lite
• Darling
• Lite Tests
• Darling Tests
• SQL collection scripts
• Documentation
• Full Dashboard (deprecated)
• CLI Installer (deprecated)

How was this tested?

See the test plan: compile-verified on all four projects (zero warnings), every live scenario driven through the real tool methods against a throwaway TimescaleDB 2.28.1 / PostgreSQL 18 container with the migrated schema; tests first execute in CI.

Checklist

• I have read the contributing guide
• My code builds with zero warnings (dotnet build -c Debug)
• I have tested my changes against at least one SQL Server version
• I have not introduced any hardcoded credentials or server names

[claude]

Reviewed the diff (DarlingTrendReader/DarlingMcpTrendTools, Lite McpQueryTools, and both instruction docs + test suites).

• Tier routing logic (ResolveTier, DescribeCoverage, DescribeEmptyCoverage, the raw/hourly branches in EmptyRoutedTrendAsync): traced the boundary cases by hand (measured vs. unmeasured coverage, past-horizon vs. not, floor beyond window end, RawRetentionApplies off on plain PostgreSQL). All consistent with the documented ladder (age → availability → comparative coverage), and the constants back it up — HourlyRetentionSpan is 90 days against MaxHoursBack = 168h, so the daily tier is correctly never needed on this ladder.
• SQL interpolation: HasAnySampleOnRouteAsync and the two *HourlySql consts interpolate TimescaleSupport.*View constants directly into the query text. Confirmed those values only ever come from the fixed TimescaleSupport constant set (never user/request input), so this isn't an injection vector — consistent with the existing pattern elsewhere in this file.
• Lite/Darling parity: both McpInstructions.cs/DarlingMcpInstructions.cs docs, the shared source/effective_start/effective_hours_back/truncated/bucket/aggregate_note envelope shape, and the TruncationSlack constant (90 min) are kept in lockstep, including duplicate pinning tests on both sides (Lite can't reference Darling's assembly, so the value is asserted independently on both — noted and appropriate).
• DateTime handling: new timestamp binding goes through DarlingMcpReadParameters.AddTimestamp/AddWindow, which already strips DateTimeKind per CONTRIBUTING.md's Postgres convention.
• Did not flag the missing-index DMV folklore (n/a here) or T-SQL style (no T-SQL collector code touched in this PR).

No correctness or parity issues found. This is unusually well covered by unit + live-Postgres integration tests for the tiering edge cases (never-sampled vs. quiet vs. unserved-head, per-grain rollup availability, tail-only server coverage).

[claude]

LGTM — reviewed tier-routing logic, empty/never-sampled/unserved-head branching, Lite/Darling envelope parity, SQL interpolation safety, and DateTimeKind handling; no substantive issues found (see inline comment for details).

[claude]

RawRetentionApplies is computed from the whole-store rollups != RollupAvailability.None, not from hourlyAvailable (the per-grain flag already passed into this method a few lines up).

The retention-arming code (EnsureRetentionPoliciesAsync / the #1680 gate in TimescaleSupport.cs) arms each raw table's retention policy independently, based on that table's own rollup coverage. If, say, only the procedure grain's rollup built successfully (QueryGrainHourly = false, some other grain true — the exact "failure-isolated partial build" state RollupAvailability's own doc comment calls out as a real, supported product state, #1664), then query_stats's retention policy is held paused — raw query_stats rows are not actually being dropped. But rollups != RollupAvailability.None is still true here (because another grain's rollup exists), so RawRetentionApplies comes back true for the query-duration route too.

Concretely: with that store shape, get_query_duration_trend for a 7-day window with a genuinely-quiet query_stats in the requested window hits EmptyRoutedTrendAsync's raw branch, computes pastRawHorizon = true, and returns the "rows were dropped… widening cannot help… run --backfill-rollups" message — even though nothing was dropped and widening (or just waiting) might actually surface data. That's the exact false-and-harmful narrative this PR's own description calls out for the pre-fix bug, just reintroduced for the partial-rollup-build edge case.

DurationTrendTierRoutingTests.AStoreWithoutTheRollup_RoutesToRaw_PerGrain pins this behavior (asserts RawRetentionApplies == true for noProcedureRollup on the procedure route), so it's a deliberate choice rather than an oversight — but it looks like it should instead be RawRetentionApplies: hourlyAvailable, matching the grain-scoped flag that's already threaded through this exact method.

[erikdarlingdata]

Fixed in
9458264: RawRetentionApplies is now the per-grain hourlyAvailable flag, exactly as you describe — the #1680 gate arms each raw table's purge against its OWN rollup's coverage, so a grain whose aggregate failed its ensure sweep keeps every raw row while the store's other grains are being dropped, and a store-wide test would have told that grain's caller the rows were gone. The record's remarks now say why it is grain-scoped, the past-horizon raw branch's "rollup not present" arm is removed (a grain with no rollup can no longer reach that branch — RawRetentionApplies is false there, so it answers quiet or "predates the store's history" from the measured raw oldest), and AStoreWithoutTheRollup_RoutesToRaw_AndKeepsRawComplete_PerGrain pins the split: on the no-procedure-rollup store the query route carries the flag and the procedure route does not.

[claude]

1. DarlingTrendReader.ResolveDurationTrendRoute sets RawRetentionApplies: rollups != RollupAvailability.None — a whole-store check — instead of using the per-grain hourlyAvailable flag already computed for this route. In the "failure-isolated partial build" state the codebase explicitly supports (#1664, e.g. one grain's rollup fails to build while others succeed), a grain whose own rollup is missing has its raw table's retention policy held paused by the #1680 arming gate — so raw is not actually being dropped for that grain. But because some other grain's rollup exists, RawRetentionApplies still comes back true, and the empty-answer path (EmptyRoutedTrendAsync) can then tell the caller the rows were dropped by retention and that widening hours_back / --backfill-rollups is the remedy, when in fact nothing was dropped and the window may just be genuinely quiet. That's the same "false and harmful" narrative class this PR sets out to eliminate, reintroduced for the partial-rollup-build edge case. See inline comment for details and a concrete repro.

Everything else checked out well: the tier-routing logic (ResolveTier/ShouldUseRawTier/coverage rungs), the Lite/Darling parity on the disclosure block (source/effective_start/effective_hours_back/truncated/bucket/aggregate_note) and shared TruncationSlack constant, the SQL (all parameterized, view names only ever interpolated from internal TimescaleSupport constants, never user input), and the extensive test coverage for the tiering ladder are all solid.

Remediated in 9458264: RawRetentionApplies is now the per-grain hourlyAvailable flag (the #1680 arming gate is per raw table), with the pin updated to assert the per-grain split. Inline reply on the finding.

[claude]

Non-blocking question: RawRetentionApplies is set to hourlyAvailable (whether the grain's CAGG object exists), but the doc comment above frames it as tracking whether the #1680 arming gate has actually armed this grain's raw purge. Per RetentionHoldReading.Armed (timescaledb_information.jobs.scheduled), a CAGG can exist while its retention policy is still held paused pending backfill — existence and "armed" aren't quite the same signal.

In practice this looks safe today: the empty-branch logic in DarlingMcpTrendTools.EmptyRoutedTrendAsync only falls back to RawRetentionApplies (via pastRawHorizon) when Coverage.RawOldestUtc is unmeasured (store-wide raw table completely empty), and that case is already intercepted earlier by the "unavailable" (never-sampled) branch in every reachable scenario I could construct. So I don't think this produces a wrong message today, but it's worth a second look (or a comment noting the approximation is deliberate) in case a future caller reads RawRetentionApplies directly for something other than this fallback.

[erikdarlingdata]

Deliberate, and now said in the record's remarks (8e40319). Existence is an UPPER BOUND on "can have been dropped": a rollup that does not exist cannot have its grain's purge armed (the gate reads that rollup's min(bucket) and holds on nothing), while a rollup that exists may or may not have armed it yet — and the flag only ever claims can, never did. That is the right bound for its one consumer: wherever raw's oldest row is measured (Coverage.RawOldestUtc), the measurement decides and the flag is not consulted; it backstops the unmeasured case only, and there it errs toward the horizon message (which still names the facts it has and the remedy) rather than toward "quiet, widen", the false direction. Reading timescaledb_information.jobs.scheduled per empty answer would spend a catalog round trip refining a fallback the measurement already makes rare — and would still not tell us whether a purge that IS armed has run over this window yet.

[claude]

LGTM — reviewed the tier-routing logic in DarlingTrendReader/DarlingMcpTrendTools and its Lite twin in McpQueryTools, cross-checked units/constants against TimescaleSupport (elapsed_time_sum, HourlyBucket, RawRetentionSpan), verified the disclosure-block shape and message text against the extensive live-TimescaleDB and DuckDB test suites, and confirmed Lite/Darling parity (both McpInstructions files, both SerializeTrend/EmptyStatus helpers, TruncationSlack pinned equal on both sides). No SQL injection surface (all new SQL is parameterized or built from internal constants), no missing-index folklore, no correctness bug found. Left one non-blocking inline question about RawRetentionApplies using rollup-existence as a proxy for the #1680 arming-gate state — traced through the fallback logic and it looks safe today, just worth a second look.

[claude]

Non-blocking parity note (pre-existing, not introduced by this PR): this disclosure block now covers the duration-trend trio on both SKUs, but get_query_trend (further down in this file) still returns the bare {server, database_name, query_hash, hours_back, data_points, trend} shape with none of source / effective_start / effective_hours_back / truncated / bucket / aggregate_note — while Darling's get_query_trend (DarlingMcpTrendTools.GetQueryTrend) has carried that six-key block since #2353. That leaves get_query_trend as the one tool where the two SKUs' envelope shapes disagree; might be worth a small follow-up to give Lite's twin the same fixed source: "raw", bucket: "per-collection" block this PR just added to its duration-trend siblings.

[erikdarlingdata]

Taken in-lane (10ba3f9) rather than filed — the file is in this lane's boundary and the helper already existed one screen up. Lite's get_query_trend data envelope now carries the same six keys between hours_back and data_points (source raw, per-collection, effective_start = the first point, truncated by the shared slack), the Lite instructions row says so, and TrendEmptyParityToolTests asserts it over the same seeded query. Its empty branch stays a bare status like Darling's get_query_trend empty branch, so the two twins now match on both paths.

[claude]

Reviewed the tier-routing logic in DarlingTrendReader.cs / DarlingMcpTrendTools.cs and its Lite twin in McpQueryTools.cs, plus the accompanying unit and live-DB tests.

What I checked:

• ResolveTier / ResolveQueryDurationTrendRoute / ResolveProcedureDurationTrendRoute correctly reuse the existing TierCoverage.Covers / ReachesFurtherBack primitives (nothing reinvented), and the age → availability → coverage ladder matches what's documented and what get_query_trend already does.
• RawRetentionApplies is correctly grain-scoped (procedure vs. query), confirmed against AStoreWithoutTheRollup_RoutesToRaw_AndKeepsRawComplete_PerGrain.
• The three-state empty-branch logic in EmptyRoutedTrendAsync (never-sampled / genuinely-quiet / unserved-head) is internally consistent for every combination of pastRawHorizon, measured RawReaches, and rollup floor I traced through.
• The Query Store sibling's move from a source object to a source string + routing sub-block is fully mirrored in QueryStoreTrendRoutingLiveTests.cs.
• Lite's McpQueryTools trio (get_query_duration_trend, get_procedure_duration_trend, get_query_store_duration_trend) picks up the identical six-key disclosure block with Lite's truth (always raw), matching Darling's shape — good parity for the tools actually changed here.
• No T-SQL in this diff (pure C# + Postgres-dialect SQL strings + tests), so the T-SQL style conventions don't apply.

One parity note (left as an inline, non-blocking comment): get_query_trend itself is not touched by this PR, and it turns out Lite's get_query_trend has never carried the source/effective_start/... disclosure block that Darling's has carried since #2353 — that's a pre-existing asymmetry, not something this PR introduces, but worth a follow-up given the PR's own stated contract that "a response-shape change here must land in Lite's Mcp*Tools too, and vice versa."

Nothing else found that looks like a correctness bug, injection risk, or parity drift introduced by this change.

[claude]

LGTM — reviewed the retention-tier routing ladder (age/availability/coverage), the three-state empty-branch messaging, the Query Store source-shape move, and Lite/Darling parity for the tools this PR actually touches; no correctness bugs, injection risks, or introduced parity drift found. One pre-existing (not introduced here) Lite/Darling shape asymmetry on get_query_trend left as a non-blocking inline note.