Skip to content

The xmin horizon alert false-fires on 1-of-1 observations and misses rotating holders #3537

Closed

Description

@erikdarlingdata

The persistence gate is identity-keyed by design (the code's rationale is written and sound: "sixty short transactions are normal, one that will not end is the incident"), but two edges escape it in opposite directions.

Evidence (verified): DarlingPostgresAlertReadAdapter.cs:88-95 (XminSql) — observations_total = COUNT(DISTINCT collection_time) over pg_xmin_horizon, and the collector emits rows only when something holds the horizon, so the denominator counts holder-bearing collections only (the doc comment at :72-76 says so deliberately); observations_held requires the SAME source+holder as the latest winner. PostgresAlertEvaluator.cs:400-401 fires at held/total >= XminPersistenceFraction.

Edge 1 (false fire): first holder after 2 quiet hours -> 1/1 = 100% -> the alert's "chronic" framing on a single observation.
Edge 2 (false quiet): a horizon continuously held past threshold by a parade of DISTINCT hourly sessions — each new holder never reaches the fraction, so the alert never fires while its own claim ("vacuum is reclaiming nothing cluster-wide") is true the whole time. The claim is about the horizon; the gate tests the holder.

Fix: ADD a horizon-persistence arm (xmin_age above threshold in >= K of the last N collections regardless of holder identity, with N as the real capture count) alongside the identity fraction, which stays for the remedy text. A minimum-observations floor closes edge 1.

From the 2026-09 brains-review campaign (alert-semantics lane, finding A7); verified first-hand at the cited lines.

Activity

  1. erikdarlingdata commented on Sep 18, 2026

    @erikdarlingdata
    OwnerAuthor

    Fixed on dev in PR #3555 (2026-09 brains-review wave). Closing per the dev-merge convention; the fix reaches main with the next release cut.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions