Self-alert FIRINGS are never logged — only resolutions are, so the log shows 'Recovered' with no preceding event #1681

Field finding from prod-sql-use2-monitor-01. Raised about the compression-job stuck alert specifically; verifying it showed the gap is broader — it affects every self-alert, not just that one.

Verified in code

DarlingSelfAlertEvaluator.RecordResolutionAsync (:963) logs at Information:

_logger?.LogInformation("[{Server}] {Title}: {Message}", resolution.ServerName, resolution.Title, resolution.Message);

The firing path has no equivalent. It calls _deliverer.DeliverAsync(new AlertOutcome(...)) (:954-958) with no logger call at all. The only _logger uses in the compression path are LogError for an exception (:671).

So the service log shows "Compression Job Recovered" with no preceding "Compression Job Stuck" — which is worse than logging neither, because it reads as a spontaneous recovery from nothing. Anyone tailing or grepping the log sees half a story.

This applies to every self-alert: disk pressure, capture-down, agent-not-running, collection-health. All fire silently and only announce themselves when they clear.

Field evidence for why it matters

9 stuck events over 6 days on that store (2026-07-21 through 07-26), always jobs 1020/1026 — the compression policies for procedure_stats and query_stats, the two highest-write-volume hypertables. No other compression job (of 37) has ever appeared. Auto-heal has worked every time, so nothing was lost — but the only record is config.config_alert_log, and you have to already suspect the problem to query it.

Fix

Log the firing where resolutions are logged, at a severity matching the alert (Warning for these), so the pair is symmetric. One change in the shared deliver path covers all self-alerts rather than special-casing compression.

Related: no TimescaleDB job audit trail exists on that store

timescaledb.enable_job_execution_logging is off (the default, never explicitly enabled), so timescaledb_information.job_errors and job_history return zero rows for every job — including jobs with dozens of confirmed successful runs. There is no per-run start/finish/success/error record for any background job, so a hung run leaves no captured error, no worker PID, nothing.

Recommend enabling it in the managed-Postgres conf append (DarlingManagedPostgres.BuildConfAppend) so the next occurrence produces evidence. Cheap, and there is no way to diagnose the hang category without it.

Hypothesis for the hang itself, explicitly unconfirmed

Given it only ever hits the two highest-throughput hypertables and never the other 35, the likely mechanism is lock/resource contention between ongoing high-volume inserts and the compression job's attempt to lock the just-closed chunk on the same table. No internal scheduler evidence exists to prove this — that is what the logging GUC above would provide. Not urgent: auto-heal is 9-for-9 with no compression backlog, but the alert text itself notes the re-arm budget is finite and will eventually escalate instead.