Postgres fleet (pgmon) has no per-target operational alerting — only tool-health and 3 outage-predictor alerts fire #2711

Summary

Part of a broader alerting-integration effort scoped 2026-08-31 (headless notification-config exposure and Datadog-parity payload structure are two sibling issues being filed in parallel).

Tonight, SQL Server per-target alerts (Deadlocks Detected/Cleared, Blocking Detected/Cleared, High CPU/CPU Resolved, Poison Wait, Long-Running Query/Cleared, Failed Agent Job) fired repeatedly and correctly on the use1/use2 fleets. The Postgres fleet (pgmon) never produced a single one of these — every pgmon alert observed was tool-health (Collection Stopped, Collector Cost Regression, Store Job Over Cadence). This is not because Postgres alerting is unwired — it's a real but narrower and differently-shaped gap than "Postgres has nothing."

Confirmed mechanism (traced, not guessed)

EvaluateAlertsAsync (Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs:2724, called unconditionally from ProcessServerSweepAsync at :1718 for every connected server every 30s, no engine check at either call site) does two things:

1. await engine.EvaluateServerAsync(snapshot, cancellationToken) — the shared AlertEngine, which evaluates Deadlocks/Blocking/CPU/Poison-Wait/Long-Running-Query/Failed-Agent-Job for every server regardless of engine. Its read adapter (DarlingAlertReadAdapter.cs) queries exclusively SQL-Server-only-populated tables:

   • Deadlocks → FROM deadlocks (:247)
   • Blocking → FROM dmv_blocking_snapshots / FROM blocked_process_reports (:106, :83, :201)
   • Poison Wait → FROM wait_stats (:296)
   • Long-Running Query → FROM query_snapshots (:356)
   • Failed Agent Job → FROM running_jobs (:710)
   • High CPU → a separate read, ReadLatestCpuAsync (DarlingWorker.cs:2865), FROM cpu_utilization_stats selecting sqlserver_cpu_utilization/other_process_cpu_utilization — column names are SQL-Server-specific.

   None of these tables are ever written by a Postgres collector (confirmed: the PG collector set uses entirely distinct table names — pg_deadlocks, pg_wait_stats, pg_kernel_stats, etc. — verified via TargetTable => on PgDeadlocksCollector.cs:102, PgKernelStatsCollector.cs:108, PgWaitStatsCollector.cs:125). So for a Postgres server_id, every one of these queries returns zero rows — there is no explicit engine gate in AlertEngine/DarlingAlertReadAdapter; the effective exclusion happens entirely because the collector-level engine gate (CollectorCatalog.AppliesTo) already stops SQL-Server-only collectors from ever writing rows for a Postgres server_id. The alert-side code doesn't know or care that it's being asked a question its data source can never answer.

2. if (runtime.Target.Engine == CollectorTargetEngine.PostgreSql) await EvaluatePostgresAlertsAsync(...) (DarlingWorker.cs:2748-2753) — a real, already-shipped, tested Postgres-specific alert path, gated correctly on engine, evaluating three "Tier 0 outage predictors" via PostgresAlertEvaluator.Evaluate (PerformanceMonitor.Alerting/PostgresAlertEvaluator.cs, tested in Darling.Tests/PostgresAlertEvaluatorTests.cs):

   • Wraparound risk (GetWraparoundRiskAsync)
   • Xmin horizon (GetXminHorizonAsync)
   • Replication slot risk (GetReplicationSlotRiskAsync)

   These three read from DarlingPostgresAlertReadAdapter.cs and deliver through the same IAlertDeliverer/mute-rule/cooldown machinery as every SQL Server alert (per-subject cooldown keying, explicitly documented at DarlingWorker.cs:2797-2799 to avoid the [FEATURE] Add a stable, machine‑readable dedup fingerprint (and involved‑object list) to alert payloads - esp. deadlocks & blocking #1140-style collapse-across-subjects bug). This is mature, tested code — it just evaluates conditions (transaction-ID wraparound, replication-slot bloat) that are meaningless for SQL Server, so Postgres is actually AHEAD of SQL Server here, not behind — these three alert types have no SQL Server equivalent at all.

So the real gap, per alert type

Where this would plug in

AlertEngine is a single shared class evaluating everything for every server with no engine gate — mirroring it directly for Postgres would mean either (a) adding Postgres-shaped read methods to the SAME IAlertReadAdapter/AlertEngine pair with each check internally branching on data availability, or (b) following the EvaluatePostgresAlertsAsync precedent already established for the three outage predictors — a sibling evaluator gated at the DarlingWorker.cs:2748 call site on CollectorTargetEngine.PostgreSql, reading from DarlingPostgresAlertReadAdapter (or a new adapter), delivering through the same IAlertDeliverer path. Given the codebase's existing engine-seam convention (CollectorTargetEngine/ITargetProvider for collectors, now EvaluatePostgresAlertsAsync for these three alerts), option (b) is the precedent already in place — worth following it rather than inventing new shared-engine branching.

Scope note

This issue is diagnosis/scoping only — no code changes. Deadlocks and Blocking look like the highest-leverage, lowest-risk additions (data already flowing, direct conceptual match). High CPU and Long-Running Query need a small design decision on shape. Poison Wait needs a from-scratch definition of what "poison" means on Postgres, if it's wanted at all.