Flaky: CompressionJobSelfHeal detection test false-fails on the nightly despite its settle-wait #1760

TimescaleSupportTests.CompressionJobSelfHeal_DetectionQueryValid_AndRearmSucceeds_AgainstDevPostgres intermittently fails on the nightly's live-Postgres job. It has already been hardened once for this exact class and failed anyway, so the existing wait is insufficient rather than absent.

Evidence

Nightly run 30234946444, dev at 8849ce97:

attempt 1   Darling PostgreSQL tests   FAILURE
  Failed  TimescaleSupportTests.CompressionJobSelfHeal_DetectionQueryValid_AndRearmSucceeds_AgainstDevPostgres [54 ms]
  Assert.DoesNotContain() Failure: Filter matched in collection

attempt 2 (rerun, NO code change)   Darling PostgreSQL tests   SUCCESS

Green on a rerun of the identical tree is what makes this a flake rather than a defect. Everything else in that run passed, including both gated store-upgrade tests.

The assertion and why it is timing-dependent

using (var arm = new NpgsqlCommand("SELECT alter_job($1::integer, next_start => now() + interval '1 hour')", connection))
{ ...; await arm.ExecuteNonQueryAsync(ct); }
await WaitForSettledNextStartAsync(connection, jobId, ct);

var healthy = await TimescaleSupport.ReadStuckCompressionJobsAsync(connection, DateTime.UtcNow, null, ct);
Assert.DoesNotContain(healthy, s => s.JobId == jobId);

The detection query reported a job as stuck that the test had just deliberately settled into a healthy state. The test's own comment documents the class:

...asserting immediately after ApplyCompressionPolicy raced that window and intermittently false-failed on a slow CI runner. Deterministically settle the job into the healthy state the assertion is actually about... then wait for the catalog to reflect a non--infinity next_start.

So WaitForSettledNextStartAsync was added precisely to close this, and the failure happened anyway. Two candidates, and they want distinguishing rather than guessing:

1. The wait returns before the state it is waiting for is actually visible — e.g. it polls only next_start while ReadStuckCompressionJobsAsync also keys off another column (last_run_status, last_finish, a job_stats row that has not materialised yet). Then the wait is satisfied while the detection query still sees a stuck-looking job.
2. The wait times out and returns quietly on a slow runner, leaving the test to assert against an unsettled catalog.

Worth checking (1) first: if the detection query reads a signal the wait does not poll, no amount of extra timeout fixes it, and the wait would need to poll the same predicate the detection query uses.

What NOT to do

Do not wrap this in a retry attribute. A retry hides both candidates above and would also hide a real regression in ReadStuckCompressionJobsAsync, which is production code behind the compression self-heal (#1585/#1586). The fix is to make the settle deterministic — ideally by having the test wait on the same predicate the detection query evaluates, so "settled" means "the thing under test agrees it is settled" rather than a proxy.

Impact

Nightly-only, and not package-blocking: the code under test was untouched by anything in the current release, the build job passed, and the assets published. But it costs a red nightly and, more expensively, a diagnosis each time — this one consumed a release-cut verification cycle before it was recognised.

Found while cutting the nightly for the client package; the compression self-heal code itself is #1585/#1586 territory and was not modified.