Forced Plan Failing re-fires on every cooldown against the same unchanged observation until the next collection lands: one failure, six cards #3579

Summary

The Forced Plan Failing alert re-fires on every cooldown expiry for as long as no new collection has arrived — so one failure event produced six alert-log rows in 28 minutes, every one reading "New Failures: 1", for a plan whose forcing failed exactly once. The condition is evaluated on every ~30-second alert pass against a fact that only changes every ~15 minutes (collection cadence), with a ~5-minute cooldown in between. "Keeps failing" and "no new data yet" are indistinguishable to the engine at pass granularity, and it treats the second as the first.

Evidence (one production store, one plan, this morning)

The raw per-collection series for the plan (force_failure_count, is_forced_plan, plan_forcing_type):

03:47   0   f   NONE
04:02   0   f   NONE
04:18   1   t   AUTO      ← automatic plan correction forced the plan; the force failed once
04:50   0   f   NONE      ← APC released the forcing; the counter honestly reset with it
05:01   0   f   NONE

The alert log for that plan: 04:18:53, 04:24:31, 04:30:04, 04:35:33, 04:40:49, 04:46:03 — six rows, all between the 04:18 and 04:50 collections, spaced at the cooldown. For those 32 minutes ForcePlanFailuresSql's newest sighting (rn=1) was the 04:18 row and its previous sighting (rn=2) was 04:02, so n.failures > p.failures was true on every pass. The cards' numbers were honest throughout — New 1 (the delta between those two sightings), Total 1 (the counter at the newest) — the defect is repetition, not arithmetic. The channel saw two of the six (webhook-side throttling), which is how it was first noticed as "two cards 17 minutes apart, totals not accumulating"; the store shows the whole shape.

Mechanism, at the code

AlertEngine.CheckForcePlanFailuresAsync documents itself as a "standing condition … while a plan keeps failing it re-fires on the cooldown" (mirroring the database-state family). That is the right contract for a condition whose input is re-measured every pass. This condition's input is a stored delta between the two newest collections, and the adapter's own doc promises "every row here is a live failure rather than accumulated history" — true at the collection instant, false for the rest of the inter-collection interval. Nothing carries the observation's identity (its collection_time) from the adapter to the engine, so the engine cannot tell a re-read of the same observation from a new one.

Fix shape

The adapter returns the observation stamp (n.collection_time) with each row, and the engine fires once per new observation that shows a rise — remembering the last alerted stamp per (server, database, query, plan) and declining to re-alert the same stamp, regardless of cooldown. A plan that keeps failing across successive collections still re-fires (each collection is a new observation with a new rise); a plan observed once fires once. Recovery semantics unchanged (the plan stops appearing → resolved).

Census before writing: any other engine condition that reads a stored between-collections delta (the "mirroring the database-state family" pointer is the first place to look) has the same shape if its cooldown is shorter than the feeding collector's cadence, and should take the same observation-keyed guard.

Detection credit: the monitoring seat's dogfood-channel read of the v3.8.0 first night. Discriminator: the raw series above, pulled with the alert log side by side.