Skip to content

The PostgreSQL Long-Running Query read applied excludedDatabases AFTER LIMIT, so an excluded database's sessions consumed the page and the alert came back short or empty while matches existed; the list is now the candidates CTE's third flag ahead of the cap, counted on the card (Closes #3742) — and the Aurora wait-profile detector gains the peak-AND-mean gate its twins have (partial #3691) - #3780

Merged
erikdarlingdata merged 1 commit into
devfrom
fix/3742-pg-lrq-excluded-databases
Sep 20, 2026

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Two PostgreSQL-target twins catch up with shapes the other seat landed today: the Long-Running Query read takes #3772's three-flag candidates CTE (closes #3742), and the Aurora wait-profile detector takes #3773's peak-AND-mean gate (partial #3691).

Item 1 — the PostgreSQL LRQ read applies excludedDatabases in the read, ahead of LIMIT $4 (Closes #3742)

The lie. DarlingPgSessionStatesReader.GetCurrentLongRunningSessionsAsync was ORDER BY query_duration_ms DESC LIMIT $4 and then, in C#, FilterExcludedDatabases dropped the rows whose database was on the shared list — and its own doc called that "the same known shape on both engines rather than a quiet divergence". It was the same shape: #3742 found it on all three reads and #3772 fixed the two SQL Server ones. Here it meant that on a server where an excluded reporting database's ETL held the five longest sessions, the page filled with rows the filter then threw away, and the alert came back short — or empty — while un-excluded long-running sessions existed. The list meant to make the alert accurate had switched it off for every other database, silently.

The fix, #3772's shape. The evaluator's ONE builder call becomes the five-argument LongRunningQueryExclusions.BuildSqlPredicates(programNameColumn, loginNameColumn, databaseNameColumn, excludedDatabases, firstParameterOrdinal) over the reader's three column constants — the new ExclusionDatabaseNameColumn = "s.database_name" beside the knob's two — so all three arms are one text in one binding order, the database operands bound after the login operands (no ordinal that existed before this arm moves; the #3736 trap walked once). The template gains {3} AS excluded_by_database in the candidates CTE, the outer WHERE NOT (program OR login OR database) ahead of LIMIT $4, and a third scalar excluded_by_database_count taken AND NOT (program OR login) — the database arm is last, so a session the knob would have removed anyway is the knob's whichever database it ran in, and the three counts sum to the sessions removed (the builder's doc says why last and not first). Exact, case-insensitive — the rule the C# filter always applied — one COALESCE(col, '') ILIKE $n ESCAPE '\' term per normalised entry, so a row with no database name is still kept, and a blank settings entry cannot bind as ''. FilterExcludedDatabases is deleted, not disabled; BuildCurrentLongRunningSessionsSql takes four arguments and has no three-argument sibling on purpose (a caller cannot forget the arm); LongRunningSessionsReadResult gains ExcludedByDatabase; the reader's excludedDatabases parameter is gone (the list arrives as rendered text + operands, like the knob, because Storage does not reference Alerting).

The receipt. DarlingWorker.EvaluatePgLongRunningQueryAsync renders AlertContextBuilders.BuildLongRunningQueryExcludedDatabasesItem — the SQL Server twin's second item, same labels (Excluded By Database / Excluded Databases) — after the knob's item, only when the list is set (empty by default, so a fresh install's card is byte-identical), and then even at 0. BuildPgLongRunningQueryExclusionDetails takes the list and the count. Annotation, never suppression: the fire is decided on the rows the read returned.

Evidence, executed here (TimescaleDB 2.28.1-pg18 on desktop-linux, torn down): the E2E plants six sessions in ReportingDb above every real one, an ETL worker in that database the knob's prefix arm also matches, six appdb sessions and one on no database. Cap 5 with the list → [400, 304, 303, 302, 301] (the no-database row and the five longest real ones), ExcludedByDatabase = 6, ExcludedByProgramPrefix = 1 (the ETL worker is the knob's); without the list the same cap holds [105, 104, 103, 102, 101] — five ReportingDb rows, the exact rows the retired shape read and threw away. Cleared knob → 7; REPORTINGDB → the same rows (case-insensitive); Reporting → nothing (exact, not prefix); a list naming every database → [400] alone, 13 excluded. Mutation: with the two-flag WHERE restored and the C# filter put back after the cap, the E2E fails Expected: [400, 304, 303, 302, 301] Actual: [] — the page went to the excluded rows and the filter emptied it. That is the issue's lie, verbatim.

Alert-path read shape: still one statement over the single latest capture, bounded by collection_time >= $3 and LIMIT $4; one more boolean column over the same rows and one more scalar over the same CTE — the SQL Server twin's cost, sub-second by shape.

Item 2 — the Aurora wait-profile detector gains the peak-AND-mean gate (partial #3691)

#3691 checklist line, verbatim: "Aurora wait-profile detector lacks the peak-AND-mean gate".

Lane 9's PgTargetAnomalyDetector.DetectWaitProfileAnomalies gated on the window peak alone — if (modifiedZ < HeavyTailModifiedZThreshold || peakRate < PgWaitProfileFallbackMsPerSec) return; — the #3538 A8 shape #3724 removed from every other baseline detector and #3773 then removed from the SQL Server wait profile: one hot five-minute delta in a quiet window fired the profile anomaly, and a longer window bought more of them. This PR copies #3773's shape: WaitRateWindowSql grows AVG(CASE WHEN interval_sec > 0 THEN … END) AS mean_ms_per_sec beside the MAX (same arm, same guard — a NULL-interval collection is NULL to both; ordinal contract 0 peak, 1 mean, 2 total, 3 samples, 4 collections); the trusted robust arm is the shared AnomalyGate.EvaluateZScore(baseline, peakRate, meanRate, HeavyTail, HeavyTail, PgWaitProfileFallbackMsPerSec, PgWaitProfileFallbackMsPerSec, SigmaDisplayCap) PAIR call (both statistics as modified z against the bucket's median and EffectiveRobustSigma; the bar on the peak only; the same 5.0 passed twice and the family's one bar as floor and fallback because the arm's EffectiveRobustSigma > 0 guard makes the classical frame unreachable); the ratio arm asks both ratios; the no-baseline arm stays on the peak's absolute bar alone, by #3741's ruling. No bar value changes; no lineage moves — the same constants, asked of one more statistic. #3773's honest caveat (the mean's z is judged against the same per-collection median/MAD — a bounded bias toward firing, never silence) is written into the summary.

Metadata stamps mean_ms_per_sec, mean_ratio, mean_modified_z beside current_ms_per_sec / ratio / modified_z (which stay the peak's) — the sampled twin's keys (lane 24, same class), not PgAnomalyDetector's avg_ms_per_sec, because the two PostgreSQL-target profiles share one vocabulary through PgTargetFactKeys.IsWaitProfileAnomaly. PgTargetScorer.IsExtremeWaitProfileAnomaly and the scorer read modified_z / ratio unchanged, so the extremity escape and the grade are what they were (pinned).

Evidence, executed here: live E2E with two Aurora servers, 31 days of one-minute pg_wait_stats cycling 100/200/300 ms/sec (bucket median 200, robust σ 148.26, trustworthy). QUIET window: 240 collections at the median, one at 3,200 (peak 20σ, mean 0.08σ) → no ANOMALY_PG_WAIT_PROFILE. SHIFTED window: at 1,500 with the same hot collection (mean 8.8σ) → one fact, current_ms_per_sec 3200, mean_ms_per_sec 1507.05, both z stamped, is_new 0, contrib_Lock:relation present, CPU excluded, no is_sampled. Red-first shown: with the gate stubbed back to peak-only, the QUIET window fires and the structural pin fails (2 red); restored, 42/42.

What it does NOT do

Tests

  • PgLongRunningQueryExclusionTests (extended, 10 unit): the database arm through the same builder call over s.database_name at $7/$8 after the login arm's $6, exact operands (no %, trimmed, _ escaped); a list with no knob binds from $5; [] and null are the same FALSE arm and the field text byte-for-byte; the three-flag WHERE ahead of LIMIT $4 (index compare) and the third AND NOT scalar; a blank database arm refused; the host's exact call and the reader's new parameter pinned (and the retired excludedDatabases: argument absent); the receipt is the twin's second item after the knob's, only when set, even at 0, alone for a cleared knob, null-guarded; README lines. Live (PgLongRunningQueryExclusionLiveTests, 2): the The PostgreSQL Long-Running Query twin ignores the program/login opt-out knob: the setting is visible, editable and read back, and does nothing on a PostgreSQL target #3743 knob fixture now also asserts ExcludedByDatabase = 0 with no list; the new excluded-database E2E above.
  • DarlingPgSessionStatesReaderTests: the stale "applied after the read" test is replaced by the third-flag pin on both renderings plus a source pin (positive control) that FilterExcludedDatabases / excludedDatabases.Any( / the reader's list parameter are gone. 33/33.
  • PgTargetAnomalyTests (+2): ungated structural + arithmetic pin (ordinal contract on the outer projection; pair call regex; inline gate gone; meanRatio on the ratio arm; peak-only is_new; the four metadata keys; The wait-profile detector judged the window peak alone: its inline gate becomes the shared AnomalyGate pair call every other baseline detector took in #3724, so one hot collection in a quiet window stops reading as a profile shift (#3741) #3773's fixture through the detector's exact AnomalyGate call — quiet no-fire, shifted fire, peak-only fires both; the escape reads the peak); the live two-server E2E above. 42/42 live.
  • DarlingPgReadSqlParsesLiveTests.NotQueryFields gains ExclusionDatabaseNameColumn with the census's idiom (3/3 live); LongRunningQueryExclusionsTests doc line that named FilterExcludedDatabases as "The PostgreSQL Long-Running Query twin ignores the program/login opt-out knob: the setting is visible, editable and read back, and does nothing on a PostgreSQL target #3743's" updated (36/36).
  • Censuses green here: RepoFileAdoptionTests, StragglerCommandTimeoutTests 48, LiveCleanupConversionRatchetTests 15, MeasurementContractCensusTests 7 (the new AVG(CASE WHEN interval_sec > 0 …) arm carries the guarded opener), FactSourceRegistryTests, PgTargetThresholdLineageTests, StoreSqlClockDisciplineTests, FleetIdentifierScrubTests, PgTargetSharedSwitchRoutingTests 27, LivePostgresCollectionHygieneTests 11, DocCommentHygieneTests 76/77 (the one red is the harness csproj in the tree — the documented artifact; it caught and I fixed one real stacked-summary insertion before push).
  • Builds: Storage, Analysis, Service (0 warnings); Darling.Tests, Lite.Tests -p:EnableWindowsTargeting=true — the only warnings are pre-existing on dev (ConfigChangeTraceAnchorTests.cs:94 xUnit2013, LogTailOverlapThresholdPinTests.cs:95 xUnit2000), none from touched files.
  • CI-first: nothing in this PR's own classes; Windows build is the first execution of the rest of the suite.

Closes #3742. Partial #3691 — "Aurora wait-profile detector lacks the peak-AND-mean gate".

Which component(s) does this affect?

  • Lite
  • Darling
  • Lite Tests
  • Darling Tests
  • SQL collection scripts
  • Documentation
  • Full Dashboard (deprecated)
  • CLI Installer (deprecated)

How was this tested?

See Tests — unit pins, both live classes, the mutation and the red-first executed on this machine against PostgreSQL 18 in Docker; CI's Windows job is the first execution of the rest of the suite.

Checklist

  • I have read the contributing guide
  • My code builds with zero warnings (dotnet build -c Debug)
  • I have tested my changes against at least one SQL Server version (PostgreSQL 18 store; no SQL Server surface touched)
  • I have not introduced any hardcoded credentials or server names

…he read ahead of LIMIT, as the candidates CTE's third flag through the shared builder's five-argument overload, counted on the card beside the knob's (#3742); the Aurora wait-profile detector gains the peak-AND-mean gate its SQL Server and sampled twins have (partial #3691)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant