pg wraparound: Warning arm signals autovacuum losing the race, not routine proximity

[erikdarlingdata]

Closes #2689.

What

The PostgreSQL Wraparound Risk Warning arm fired at 0.9 × autovacuum_freeze_max_age unconditionally — a routine operating point every healthy database climbs to on every freeze cycle before autovacuum resets it (the expected sawtooth). One fleet database re-fired the identical alert every ~5-minute collection cycle at 9% of the way to actual wraparound, while the read tool's own severity read ok.

The collector already computes what's needed to tell "routine" from "at risk" apart — freezing_is_keeping_up (has the age come back down from its own recent peak) — but the alert never consulted it.

Fix

• The relative Warning arm moves from 0.9× to 1.0× autovacuum_freeze_max_age (the crossing point where autovacuum itself forces a vacuum) and is now gated by FreezingIsKeepingUp, computed per-counter (XID and MultiXact independently, since one can be healthy while the other is stuck). A database above its setting but resetting every cycle stays silent; one stuck at or above it, never seen lower in the window, warns.
• Added an absolute early-Warning arm at 50% of the true 2^31 ceiling (WraparoundWarningFractionOfCeiling), mirroring the existing Critical ceiling arm (0.745) — on a cluster tuned high enough that the relative arm becomes unreachable, this still gives an early signal instead of jumping straight to Critical with none.
• Critical arms (2× setting, 74.5% of ceiling) are unchanged.

FreezingIsKeepingUp is plumbed from the collector's own window-peak data: DarlingPostgresAlertReadAdapter.WraparoundSql now selects MAX(frozen_xid_age)/MAX(min_multixid_age) OVER (PARTITION BY database_name) (the same technique DarlingPgWraparoundReader already uses for the read tool), carried into PostgresWraparoundAlertInfo as WindowPeakXidAge/WindowPeakMultiXactAge, with the derivation (age < windowPeak) mirroring DarlingMcpPgWraparoundTools' own formula exactly — so the alert and get_pg_wraparound_risk can never disagree about what "keeping up" means again.

Tests

• WraparoundGradesAtTheDocumentedBoundaries updated for the new 1.0× boundary (was 0.9×).
• New: WarningIsSuppressedWhileTheCounterIsComingBackDownFromItsPeak / WarningFiresWhenTheCounterHasNeverComeBackDownInTheWindow — the core regression pair proving the gate.
• New: WarningIsReachableOnAClusterTunedPastHalfTheWall — the absolute-arm counterpart to the existing Critical-ceiling test.
• FindingsAreOrderedWorstFirst bumped its Warning fixture from 190M to 200M (the new boundary).

Verified compiling (PerformanceMonitor.Alerting, PerformanceMonitor.Darling.Service, Darling.Tests all build clean on macOS via EnableWindowsTargeting); the test assembly is net10.0-windows, so xunit execution runs in CI per house convention.

🤖 Generated with Claude Code

[claude]

Reviewed the wraparound Warning-arm change (PostgresAlertEvaluator.cs, PostgresAlertInfo.cs, DarlingPostgresAlertReadAdapter.cs, and the test file).

Overall: solid. Traced through the boundary math (relative 1.0x/2.0x arms, the new absolute 0.5/0.745 ceiling arms, and the per-counter FreezingIsKeepingUp gate) against all the new/changed test cases and it checks out, including the tuned-cluster case where the relative arm becomes unreachable. The MAX(...) OVER (PARTITION BY database_name) window-peak SQL added to DarlingPostgresAlertReadAdapter.WraparoundSql is computed before DISTINCT ON collapses to the latest row (confirmed: PostgreSQL evaluates window functions before DISTINCT), so it correctly sees the whole freshness window rather than just the surviving row — and it mirrors DarlingPgWraparoundReader.PgWraparoundSql and DarlingMcpPgWraparoundTools's own FrozenXidAge < WindowPeakFrozenXidAge formula exactly, as the PR description claims. No SQL injection risk (parameterized $1/$2), no secrets, and this is Darling-only PostgreSQL functionality with no Lite counterpart to drift out of parity (Lite has no PostgreSQL support at all).

One minor nit, not blockable, couldn't leave it inline since the line falls outside the diff hunk:

Darling/Darling.Tests/PostgresAlertEvaluatorTests.cs, line 52 (WraparoundThresholdsScaleToTheServersOwnFreezeMaxAge, untouched by this PR):

/* ...and does not fire at all on one tuned to 1.5 billion, where 90% of the setting is 1.35B. */

This is now stale — the Warning arm moved from 90% to 100% of the setting in this PR (WraparoundWarningFractionOfFreezeMaxAge = 1.0), so the comment should read "100% of the setting is 1.5B". The assertion still passes either way (400M is well under both figures), which is why it wasn't caught, but given this codebase's convention of comments carrying the actual numbers/reasoning, this one now contradicts the constant it's describing.

[claude]

Correctness: the unconditional ceiling-Warning arm reintroduces the exact bug this PR fixes, for any cluster tuned above ~1.07B.

autovacuum_freeze_max_age's legal range in PostgreSQL is [100000, 2000000000] — the max is 2 billion, which is always below WraparoundCeiling (2,147,483,648). That means the relative Warning arm (age >= setting, now 1.0x) is reachable for every legal setting; unlike the relative Critical arm (2x setting), it never actually becomes unreachable in practice. So the stated rationale for this arm ("a cluster tuned high enough that the relative arm becomes unreachable, this still gives an early signal instead of jumping straight to Critical with none") doesn't hold — the only test exercising it (WarningIsReachableOnAClusterTunedPastHalfTheWall) uses freezeMax: 2_200_000_000, which is itself outside Postgres's legal range and only "proves" the arm fires in a configuration that can't occur.

The actual effect: for any legal tuning where setting > ~1.074B (e.g. 1.2B, 1.5B — the same range the pre-existing WraparoundThresholdsScaleToTheServersOwnFreezeMaxAge test already treats as a realistic tuning), ceilingWarnAt (~1.074B) sits below warnAt (= setting). A healthy database's routine sawtooth climbs to just under its own setting every cycle before autovacuum resets it — exactly the "routine, not a risk" case #2689 is about. On such a cluster the age will cross ceilingWarnAt on every single healthy cycle, long before it ever reaches its own setting, and this branch fires unconditionally (no FreezingIsKeepingUp check). That's the identical "pages on every healthy database" failure mode the PR's headline fix (gating the relative arm on FreezingIsKeepingUp) was written to eliminate — just moved to a different (but real, legal, and already-acknowledged-as-realistic-elsewhere-in-this-test-file) band of configurations.

As a bonus, the generated message text compounds this: for e.g. setting = 1.2B, age = 1.1B, the Warning body says "already well past the routine vacuum-forcing point on this cluster's own autovacuum_freeze_max_age" — but the age hasn't even reached the setting (1.2B) yet.

Worth either gating this arm on FreezingIsKeepingUp too (like the relative arm), or picking a fraction that's actually only reachable when the relative arm is genuinely unreachable (i.e. never, for legal settings, unless the ceiling arm is meant to future-proof against a raised PostgreSQL max — in which case that reasoning belongs in the doc comment).

[claude]

Review summary

Scope: PostgresAlertEvaluator.cs, PostgresAlertInfo.cs, DarlingPostgresAlertReadAdapter.cs, tests, and CHANGELOG — the PostgreSQL wraparound Warning-arm rework (#2689). This is Darling/PostgreSQL-only logic (PerformanceMonitor.Alerting is shared but IPostgresAlertReadAdapter has no Lite/SQL Server implementation), so there's no Lite/Darling parity drift to flag here.

Left one correctness finding inline on PostgresAlertEvaluator.cs: the new unconditional absolute Warning arm (WraparoundWarningFractionOfCeiling = 0.5, checked before the gated relative arm) fires regardless of FreezingIsKeepingUp for any autovacuum_freeze_max_age tuned above ~1.074B — which is still well within PostgreSQL's legal range (max 2B). Since the relative Warning arm (1.0x setting) never actually becomes unreachable for a legal setting (unlike the 2x relative Critical arm, which genuinely does), a healthy database tuned in that range will cross the absolute floor on every routine sawtooth cycle before it even reaches its own setting — reproducing the exact "pages on every healthy cycle" bug this PR sets out to fix, just for a different (real, legal) band of configurations. Details and a concrete repro in the inline comment.

Everything else checked out:

• The Grade gating logic (relative Warning gated per-counter on FreezingIsKeepingUp, absolute arms unconditional and checked first) is otherwise sound, and matches the severity-precedence ordering (Critical arms before Warning arms).
• XidFreezingIsKeepingUp/MultiXactFreezingIsKeepingUp (age < windowPeak) exactly mirrors DarlingMcpPgWraparoundTools' existing freezing_is_keeping_up derivation, as claimed.
• The WraparoundSql window-function addition (MAX(...) OVER (PARTITION BY database_name) alongside DISTINCT ON (database_name)) is valid Postgres, parameterized (no injection risk), and computes the peak over the same freshness-filtered rows the "current" row comes from, so the two can't disagree.
• Reader ordinal mapping (5/6 → WindowPeakXidAge/WindowPeakMultiXactAge) lines up correctly with the new SQL columns and the record's positional constructor.
• New tests correctly pin the 1.0x boundary and the suppression/firing pair for the gate; the "tuned past half the wall" test uses freezeMax: 2_200_000_000, which is actually part of the problem above — that value is outside Postgres's legal 2B ceiling, so it doesn't exercise a real-world configuration.
• CHANGELOG entry follows the existing style/format.