Log self-alert firings, and enable TimescaleDB job audit logging (#1681)

[erikdarlingdata]

Closes #1681. Two halves of the same field finding from prod-sql-use2-monitor-01: a compression job stuck 9 times in 6 days, and neither the service log nor the database could say anything useful about it.

Half 1 — the log showed recoveries but never firings

RecordResolutionAsync had always logged at Information. The firing path called _deliverer.DeliverAsync with no logger call at all. So the service log showed:

Compression Job Recovered

with nothing before it — a spontaneous recovery from a condition that never appeared. That is worse than logging neither half, because it reads as noise rather than a missing pair.

Verifying it showed the gap is not compression-specific. Every self-alert went through that same silent path: disk pressure, capture-down, agent-not-running, collection-health. All of them fire silently and only announce themselves on the way out.

Firings now log at Warning — a firing self-alert means something is wrong with the monitor itself, and it has to stand out from the Information-level recovery it will eventually pair with.

Muted alerts log too, flagged [muted]. Muting suppresses the notification channels, not the operator's ability to find the event afterwards; suppressing both would make a muted condition invisible everywhere at once.

Half 2 — there was no job audit trail to diagnose from

timescaledb.enable_job_execution_logging is off by default. On the field store that meant timescaledb_information.job_errors and job_history returned zero rows for every job — including jobs with dozens of confirmed successful runs. No captured error, no worker PID, no start/finish record.

That is why the hang still has no confirmed root cause, and why the next occurrence would have been equally opaque. Now enabled in the managed conf append. Rows are written once per job run, not per row processed.

What this does not do

It does not fix the hang. The working hypothesis — lock contention between high-volume inserts and the compression job's attempt to lock the just-closed chunk on the same table, given it only ever hits the two highest-write hypertables and never the other 35 — remains unconfirmed, because the evidence to confirm it did not exist. Half 2 is what makes the next occurrence diagnosable.

Auto-heal is 9-for-9 with no compression backlog, so this was not urgent; the alert text itself notes the re-arm budget is finite and eventually escalates instead.

Test plan

The harness now passes a real capturing logger to every self-alert test rather than null, so the added logging is exercised by the whole existing suite rather than only by the two new assertions — which cover that a firing logs at Warning, and that a muted firing still logs, flagged.

• Darling.Tests: 3129 passed, 0 failed (149 gated-live skipped)
• Solution builds 0 warnings, 0 errors

🤖 Generated with Claude Code