Skip to content

A baseline series can now be scoped to ONE statement: the per-key seam on PgBaselineProvider (a keyed GetBaselineAsync overload, a third resolve seam, $7 after the Q6 clock) and two per-queryid arms — a statement's share of the collection's execution time and its per-call mean — so a PostgreSQL bad actor can be judged against its OWN hour-of-week habit; no consumer yet (partial #3691, lane 33) - #3810

Merged
erikdarlingdata merged 1 commit into
devfrom
feat/3691-per-key-baseline-seam
Sep 20, 2026

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

What does this PR do?

Headline mechanism. A baseline series can now be scoped to ONE member of a population. PgBaselineProvider.GetBaselineAsync gains a five-argument overload — (serverId, metricName, string? key, analysisTime, ct) — that keys a hour×day-of-week bucket map on (server, metric, key) instead of (server, metric). The key is resolved through a THIRD seam, ResolveKeyedBaselineQuery(metricName) (protected virtual, default null — the SQL Server store declares no keyed metric), and bound as $7 after the six the compute always bound ($1 server, $2/$3 window, $4..$6 the Q6 clock). The PostgreSQL provider overrides the seam for the statement family: pg_statement_share (NEW MetricNames const — one statement's share of the collection's total execution time, from pg_statement_stats' stored deltas, per collection) and pg_statement_mean_ms keyed (one statement's per-call mean, beside lane 27's server-wide UNKEYED arm of the same name, which is untouched). Nothing consumes the seam yet — the pins are its only callers. PARTIAL for #3691 — lane 33. The checklist line, verbatim: "RULED 2026-09-20 → lanes 33 (per-key baseline seam) + 34: re-grade the bad-actor share as deviation from the statement's OWN hour-of-week share baseline, absolute share as context (derive-not-constant; the percentile raise rejected)." This PR is the lane-33 half; do not close.

Root cause

Lane 27's out-of-lane item (#3798): "per-key baseline seam: PgBaselineProvider.GetBaselineAsync keys (server, metric) only; a per-queryid series needs a key dimension." The ruling above makes it load-bearing — a share graded as deviation from the statement's OWN hour-of-week habit is impossible with one series per (server, metric): a 40% statement at its usual 40% is the workload; a 40% statement that usually takes 4% is the story, and only a per-statement series can tell them apart.

What changed, file by file

  • PgBaselineProvider.cs (root, shared): the four-argument overload is now => GetBaselineAsync(serverId, metricName, key: null, analysisTime, cancellationToken) — one lookup body, every one of the 36 existing call sites resolves to it unchanged (a DateTime and a string in the third position cannot be confused). CacheKeyFor(server, metric, key): {server}:{metric} unkeyed (the exact string it has always been, so InvalidateCache's prefix sweep keeps finding it), {server}:{metric}:{key} keyed. ComputeBaselinesAsync resolves key is null ? ResolveBaselineQuery : ResolveKeyedBaselineQuery — two seams, never one: a keyed call for a metric with no keyed shape is "no baseline", never the population's buckets under a member's key, and the reverse. ComputeBucketsAsync binds $7 only when key is not null, after $6; the successor/legacy supply swap (ChooseSupplyAsync, the SQL Server pair's) is skipped on a keyed compute — it compares against the unkeyed successor text and could never match. CachedBaseline.Key for the count.
  • Cardinality honesty. A keyed series is one cache entry and one 30-day scan per (server, metric, key) per CacheTtl. The provider refuses nothing (it cannot know which keys matter) — the CONSUMER bounds the population to its window's top-N and its doc says so (lane 34 asks for the top-share candidates the bad-actor read already returns). What the provider does: above KeyedBaselineCacheWarnCount = 500 keyed entries it logs a Warning with the count, once per cache period (ShouldWarnKeyedCardinality, pure — the closest thing the class has to "once per pass"; inside one TTL a pass's repeats are cache hits). Unmeasured, stated in the six lines above the constant: 500 is ten servers × a fifty-statement top-N at the widest consumer the campaign has ruled on; calibrate against the keyed entry counts a fleet pass actually reaches. Entries, not successes: a no-arm or failed keyed compute caches its empty like an unkeyed one does and counts.
  • PgTargetBaselineProvider.cs (root): GetPgTargetKeyedBaselineQuery(metricName) (internal, pinnable) → the two statement arms; protected override ResolveKeyedBaselineQuery. Class doc says three seams.
  • PgTargetBaselineProvider.Statements.cs (NEW partial — file-disjoint from lanes 30/32/35's partials): the two keyed arms. Both are Q6-contract arms: WITH per_collection … clean(collection_time, v) + the ONE RobustTierScaffold, server_id = $1 AND collection_time >= $2 AND collection_time < $3, one row per collection, stored deltas (delta_total_exec_time_ms, delta_calls — lane 7's discipline, no LAG), ::DOUBLE PRECISION, plus queryid = $7::BIGINT — the key is bound as text (the seam is dimension-neutral) and the arm casts the parameter once rather than the column per row (idx_pg_statement_stats_time is (server_id, collection_time); the statement predicate is a heap filter either way, so $7::BIGINT costs one cast, queryid::text = $7 tens of millions). A non-numeric key fails the cast inside the one classified catch → Empty, the same posture as any failed compute. Share: CAST(SUM(ms) FILTER (WHERE queryid = $7::BIGINT) AS DOUBLE PRECISION) / SUM(ms), HAVING SUM(ms) > 0 (a collection where nothing ran is no sample); coalesce(stmt_ms, 0) — a collection where OTHER statements ran and this one did not IS a sample at 0 (the statement's habit includes its idle hours). No unkeyed arm: the server's share of itself is 1.0. Mean ms: lane 27's arm text plus exactly one line, AND queryid = $7::BIGINT (pinned: strip that line and the two texts are byte-equal), so the per-call quantity, the numerator, the denominator and the no-calls rule are the same.
  • MetricNames.cs (shared): PgStatementShare = "pg_statement_share"; the PgStatementMeanMs doc now says two arms answer to the name.

The lineage shape of every bar

One bar: KeyedBaselineCacheWarnCount = 500, unmeasured (comment within six lines names the calibrating read). It grades nothing and stamps no fact — it gates a log line. No scorer file is touched, so PgTargetThresholdLineageTests / PgTargetMeasuredLineageTests have nothing new to read.

Co-fire gating

None — no fact, no detector, no advice. The seam is plumbing.

Censuses moved deliberately (each edited, none deleted, each red-first here)

  • LocalClockBucketKeyTests.EveryBucketStatement_KeysOnTheLocalClock_AndNoneOnBareCollectionTime — two-armed, never loosened (the Q6 owner's rule): EveryBucketStatement() now yields the keyed map's statements flagged Keyed = true by the ARM that returned them; an unkeyed statement's referenced parameter set must be exactly {1..6}, a keyed one's exactly {1..7} — exact sets by regex \$(\d+), replacing the old Contains("$3")… (which $30 would have satisfied). Floors: ≥ 17 unkeyed, ≥ 2 keyed. Keyed arms must carry queryid = $7::BIGINT and have no SQL Server arm. Red-first by mutation: dropping $7 from the share arm → "is keyed and references $1..$6 — expected exactly $1..$7"; adding $7 to lane 27's unkeyed arm → "is unkeyed and references $1..$7 — expected exactly $1..$6". Both reverted.
  • BaselineSupplyTests.PgTargetBaselineSources — the retention-floor derivation sweeps BOTH maps (a keyed-only metric's table would otherwise never reach the derivation); the eleven-table pin is unchanged because pg_statement_stats was floored by lane 27. New TheKeyedArms_ReadAFlooredTable_AndTheDerivationSweepsThem.
  • PgTargetClockTests.ThePostgresProvider_RedeclaresExactlyTheThreeSeams_AndNoState (was …TwoSeams…): 2 → 3 overrides, ResolveKeyedBaselineQuery named; still no instance state on the derived class.
  • DarlingAnomalyBaselineTests.BaselineProvider_CarriesLitesSurface… and SharedBaselineModelPinTests.PgBaselineProvider_BindsToTheSharedBaselineBucket: GetMethod("GetBaselineAsync") is ambiguous with two overloads (throws, not fails) — asked by signature; the shared-bucket pin now covers BOTH overloads (a keyed series returns the same shared bucket shape, never a keyed variant).
  • PgTargetAnomalyTests — one comment line ("exactly two overrides" → three). No assertion changed.
  • Green, no edit: PgTargetFactCollectorTests, PgTargetSharedSwitchRoutingTests, PgTargetThresholdLineageTests, PgTargetMeasuredLineageTests, StoreSqlClockDisciplineTests, DocCommentHygieneTests, RepoFileAdoptionTests (the new test uses ReadRepoFile, single-line anchors — no s_lfReaders entry), CommentFilterAdoptionTests, LivePostgresCollectionHygieneTests, LiveCleanupConversionRatchetTests, FleetIdentifierScrubTests, AnalysisShutdownResidueTests (still exactly one catch), BaselineTierJoinShapeTests, MeasurementContractCensusTests, FactSourceRegistryTests, AsOfWindowAnchorTests, BaselineTimeoutIsNamedTests, PgTargetPlanTests (lane 27's arm pins), PgTargetV3PlumbingTests, PgTargetBetweenWavesV2Tests. Lite's LocalClockBucketKeyTests counts Lite's eleven arms — Lite's provider answers null for the new name. No file mentions an aurora_stat_* function.

What it deliberately does NOT do

  • No consumer. No detector, scorer, advice, graph edge or fact reads the keyed overload. Lane 34 (the share) and lane 27's follow-up (switching ANOMALY_PG_PLAN_REGRESSION to the statement's own series) are the consumers; the seam exists so they can be written without touching the shared roots.
  • No key travels into BaselineBucket, BaselineMath or AnomalyGate — the bucket a keyed call returns is the shape an unkeyed one returns; the caller that passed the key knows what it belongs to.
  • No migration rung (V135 unchanged); no new table; pg_statement_stats was already in BaselineServingRawCollectors (lane 27).
  • Lite's BaselineProvider twin gets no keyed seam: Lite monitors SQL Server only and has no keyed consumer; the seam is engine-neutral in shape (a string? key an arm casts) and can be ported the day Lite needs it.
  • The SQL Server pass is byte-identical at the wire: the four-argument overload binds exactly the six parameters it bound before; ResolveKeyedBaselineQuery is never called on an unkeyed path; the routing pins pass unchanged.
  • The provider bounds nothing; it reports. A cap would have to pick which statements matter, and only the consumer knows.

Test plan

New Darling/Darling.Tests/PgBaselineProviderKeyedTests.cs (7 unit + 1 gated live): exactly two public overloads with the pinned signatures, the delegation spelled key: null, one lookup body; CacheKeyFor byte-identity and third segment ("" is a key; null is the only "no key"); the seam protected virtual, null for every declared metric on the base, a keyed compute resolving through it alone; the $7 bind after $6, once, under key is not null, the supply swap under key is null, one classified catch; the PostgreSQL provider's keyed set is exactly {pg_statement_mean_ms, pg_statement_share}, the share has no unkeyed arm, lane 27's unkeyed arm has no $7; both arms' contract and the mean arm's byte-equality with lane 27's modulo the one predicate line; the cardinality note's arithmetic (at the bar no, over it once, again a TTL later) and the unmeasured lineage. Live (DARLING_TEST_PG, live-postgres collection, LiveStoreCleanup): two statements over three Tuesdays × twelve five-minute collections — A 100 calls × 8 ms, B 50 × 4 ms, B absent from each Tuesday's first collection — through the REAL PgTargetBaselineProvider: share A (12, Tue) Full, 36 samples, 3 days, mean (33×0.8 + 3×1.0)/36, median 0.8; share B 36 samples (the sat-out collections are ZERO samples), mean (33×0.2)/36, median 0.2; mean-ms A 36 samples at 8.0, B 33 samples at 4.0 (sat-out collections are NO sample); lane 27's unkeyed arm still answers the server-wide (33×6.667 + 3×8)/36; the unkeyed share call, a keyed pg_tps call, and a non-numeric key are all Empty; the SQL Server PgBaselineProvider over the same store answers Empty to every keyed call; keyed entry count 6 / 2, unchanged by a cache hit, 0 after InvalidateCache. This live test is also the parse check for both arms — DarlingPgReadSqlParsesLiveTests sweeps DarlingPg*Reader types, not the providers.

Executed here, not only compiled: a net10.0 harness (AssemblyName=Darling.Tests, inside the worktree under obj/, deleted before commit) compiling every PgTarget*.cs test plus the censuses above, DarlingAnomalyBaselineTests, SharedBaselineModelPinTests, TsqlConventionGuardTests, DarlingPgReadSqlParsesLiveTests — 893 / 893 green, 0 skipped, against a throwaway timescale/timescaledb:2.28.1-pg18 (Docker context desktop-linux, role and database darling, migrated by PgMigrations.MigrateAsync, torn down after), every gated live test in the set included. The harness caught two of my own first-draft defects (a Between-slice that included its own anchor; an entry count that forgot a no-arm keyed compute is cached too — the second is now the doc). The one red at my base (TsqlConventionGuardTests.TheMemberScan…) was dev's own at 291e15f (fixed by #3806); rebased onto origin/dev at 0bcbd50, re-run 893/893. First Windows execution is CI's.

Builds, zero warnings: PerformanceMonitor.Analysis, PerformanceMonitor.Darling.Analysis, PerformanceMonitor.Darling.Service, Darling.Tests, Lite.Tests, deprecated/Dashboard.Tests (-p:EnableWindowsTargeting=true on the test projects).

Which component(s) does this affect?

  • Lite
  • Darling
  • Lite Tests
  • Darling Tests
  • SQL collection scripts
  • Documentation
  • Full Dashboard (deprecated)
  • CLI Installer (deprecated)

How was this tested?

Above. No production store or fleet server was read; the rig was a local container, torn down.

Checklist

  • I have read the contributing guide
  • My code builds with zero warnings (dotnet build -c Debug)
  • I have tested my changes against at least one SQL Server version (not applicable — the SQL Server pass binds exactly what it bound; the base declares no keyed metric and the routing pins pass unchanged)
  • I have not introduced any hardcoded credentials or server names

Partial for #3691 (lane 33). CHANGELOG deliberately untouched — the coordinator splices this wave's entries.

…BaselineProvider.GetBaselineAsync gains a keyed overload (server, metric, key) resolved through a third seam and bound as $7 after the Q6 clock, the PostgreSQL provider declares two per-queryid statement arms (share of the collection's exec time, per-call mean), and the local-clock census becomes two-armed - unkeyed text exactly $1..$6, keyed exactly $1..$7 (#3691 lane 33)

The unkeyed four-argument overload delegates with key: null and every existing caller and pin is byte-identical; the SQL Server store declares no keyed metric. Nothing consumes the seam yet - lane 34 (the bad-actor share re-graded as deviation from the statement's own hour-of-week share, the 2026-09-20 ruling) and lane 27's follow-up do.

Censuses moved deliberately: LocalClockBucketKeyTests (two-armed, exact parameter sets), BaselineSupplyTests (the derivation sweeps the keyed map), PgTargetClockTests (three seams), DarlingAnomalyBaselineTests and SharedBaselineModelPinTests (GetMethod by signature - two overloads).
@erikdarlingdata
erikdarlingdata merged commit bb7198f into dev Sep 20, 2026
6 of 7 checks passed
@erikdarlingdata
erikdarlingdata deleted the feat/3691-per-key-baseline-seam branch September 20, 2026 17:23
This was referenced Sep 20, 2026
erikdarlingdata added a commit that referenced this pull request Sep 21, 2026
…-week per-call mean (the keyed series #3810 made possible) and keeps the server-wide mean only as the cold fallback; the third calibration read's lineage lands on the bad-actor and statement-mean multiples (#3691 lane 39)
erikdarlingdata added a commit that referenced this pull request Sep 21, 2026
…-week per-call mean (the keyed series #3810 made possible) and keeps the server-wide mean only as the cold fallback; the third calibration read's lineage lands on the bad-actor and statement-mean multiples (#3691 lane 39) (#3845)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant