Bound the supervisor's stand-off for a self-managed retry and require it to be fresh (#612)

docs/BACKLOG.md

@@ -44,7 +44,7 @@ Three lanes, at most one item in flight per lane. Lane C is the owner's and is t
 | 1 | C | Acceptance session | Cut a beta from `main` and run [`docs/qa/installed-acceptance-checklist.md`](qa/installed-acceptance-checklist.md) once. Covers [#732](https://github.com/iamfatness/CoreVideoPro/issues/732), [#725](https://github.com/iamfatness/CoreVideoPro/issues/725), [#724](https://github.com/iamfatness/CoreVideoPro/issues/724), [#735](https://github.com/iamfatness/CoreVideoPro/issues/735), [#668](https://github.com/iamfatness/CoreVideoPro/issues/668), [#674](https://github.com/iamfatness/CoreVideoPro/issues/674), [#608](https://github.com/iamfatness/CoreVideoPro/issues/608), [#582](https://github.com/iamfatness/CoreVideoPro/issues/582), [#568](https://github.com/iamfatness/CoreVideoPro/issues/568), [#569](https://github.com/iamfatness/CoreVideoPro/issues/569), [#581](https://github.com/iamfatness/CoreVideoPro/issues/581), [#473](https://github.com/iamfatness/CoreVideoPro/issues/473) and the first live run of the 2026-10-01/02 fixes ([#740](https://github.com/iamfatness/CoreVideoPro/issues/740), [#739](https://github.com/iamfatness/CoreVideoPro/issues/739), [#757](https://github.com/iamfatness/CoreVideoPro/issues/757), [#758](https://github.com/iamfatness/CoreVideoPro/issues/758), [#759](https://github.com/iamfatness/CoreVideoPro/issues/759), [#760](https://github.com/iamfatness/CoreVideoPro/issues/760)). Record PASS, FAIL or `MISSING_EVIDENCE` per row on its issue. |
 | 2 | B | [#762](https://github.com/iamfatness/CoreVideoPro/issues/762), [#741](https://github.com/iamfatness/CoreVideoPro/issues/741) | Merged: the shell refuses Engine on, Record or Join from the core profile with a named reason, and the core refuses Record on a failed encoder and Join with no engine (`docs/reference/core-on-air-policy.md`). Remaining: first installed run. |
 | 3 | B | [#754](https://github.com/iamfatness/CoreVideoPro/issues/754) | Three intermittent failures in `native-shell-windows`. Rule out a real double-rejoin behind `AStartDuringCrashRecoveryStillRejoinsZoom`; make the tests deterministic. |
-| 4 | A | [#538](https://github.com/iamfatness/CoreVideoPro/issues/538) with [#703](https://github.com/iamfatness/CoreVideoPro/issues/703) | Owner's top code priority (2026-09-29): one shared hardware Program encode for RTMP/SRT/HLS, compositor-clock video, sample-counted audio, bounded mux queues, decoded sync evidence. Taken with it: YouTube audio lag ([#703](https://github.com/iamfatness/CoreVideoPro/issues/703)); the keyframe interval (#605) was already applied in #707 and the RTMP codec overwrite (#606) is fixed. Then [#612](https://github.com/iamfatness/CoreVideoPro/issues/612). |
+| 4 | A | [#538](https://github.com/iamfatness/CoreVideoPro/issues/538) | Owner's top code priority (2026-09-29): one shared hardware Program encode for RTMP/SRT/HLS, compositor-clock video, sample-counted audio, bounded mux queues, decoded sync evidence. #703 (YouTube audio lag) closed as resolved by the owner 2026-10-02; #605, #606, #602 and #612 are done. Remaining is the parent: shared hardware encode evidence and decoded sync across destinations. |
 | 5 | B | [#701](https://github.com/iamfatness/CoreVideoPro/issues/701), [#624](https://github.com/iamfatness/CoreVideoPro/issues/624), [#705](https://github.com/iamfatness/CoreVideoPro/issues/705), [#750](https://github.com/iamfatness/CoreVideoPro/issues/750) | Live-show media defects: media playout stutter or freeze, a subscribed Zoom feed that stops advancing in Tiles, headless media audio missing from Program, and the 13 to 18 ms media audio offset ([#750](https://github.com/iamfatness/CoreVideoPro/issues/750) needs an owner ruling). |
 | 6 | C | [#657](https://github.com/iamfatness/CoreVideoPro/issues/657) | Qualify the merged control-state slices in an installed live meeting; folded into the acceptance session. If real guest transitions are unavailable, record `MISSING_EVIDENCE`. |
 | 7 | A | [#536](https://github.com/iamfatness/CoreVideoPro/issues/536) | Finish honest RTT and codec/packet decode-error health, then validate failure and recovery. Ingest status now reaches the UI ([#760](https://github.com/iamfatness/CoreVideoPro/issues/760)). |
@@ -86,7 +86,6 @@ These can preempt Next on a show night. They do not jump the ranked Now item unt
 | [#674](https://github.com/iamfatness/CoreVideoPro/issues/674) | Slices A–F merged (#675–#680), including direct negotiated-format facts and existing Multiview texture crop. Installed no-click video-off, ISO arm, Engine-off transitions and ten-row GPU load remain acceptance; reopened after an early close. |
 | [#473](https://github.com/iamfatness/CoreVideoPro/issues/473) | Installed beta ProRes media decoder shows a placeholder while the same code works in a dev launch; reproduce and fix the installed path. |
 | [#652](https://github.com/iamfatness/CoreVideoPro/issues/652) | Matched A/B found GoXLR live skew near −240 ms on both accepted beta and prior main, while recorded Program stayed near −20 ms. #655 reduced live skew to −46.2 ms with zero underruns/lost samples over 24 s; real-human and long-show drift acceptance remain. |
-| [#703](https://github.com/iamfatness/CoreVideoPro/issues/703) | Saved YouTube replay has about 0.7 s audio lag. Same-run Program/RTMP local pattern is near one frame with #705's media PCM fix; the sender reports video and audio on the saved YouTube test destination. Fresh YouTube playback measurement and frame-delivery acceptance remain missing. |
 | [#649](https://github.com/iamfatness/CoreVideoPro/issues/649) | RTMP demuxer test compares `const char*` literal addresses; use content comparison so Debug native gate is deterministic. |
 | [#615](https://github.com/iamfatness/CoreVideoPro/issues/615) | RTMP packet amplification is measured and coalesced. Encoder frames that never reached the shed counter are retained (main `61e83ed1`). The original PC-wide outage and full-show continuity still need fleet acceptance. |
 | [#627](https://github.com/iamfatness/CoreVideoPro/issues/627) | Per-participant audio delay. Capture devices already have `setAudioSyncOffset`; a Zoom guest does not. Headless clap on the Zoom-audio mailbox build measured one pipeline skew (median -39.0 ms, audio lags video), not a per-person offset. |
@@ -144,7 +143,6 @@ the sole work list. Their order in this table is not a priority assignment.
 
 | Issue | Remaining work |
 |---|---|
-| [#612](https://github.com/iamfatness/CoreVideoPro/issues/612) | Sender supervision: deterministic give-up during backoff (the stand-off is keyed on a latched result code with no freshness test or deadline). |
 | [#607](https://github.com/iamfatness/CoreVideoPro/issues/607) | Encoder/sender correctness: the bitstream queue caps are memory bounds used as a latency bound; give it a measured latency bound. |
 | [#611](https://github.com/iamfatness/CoreVideoPro/issues/611) | Move test-only virtual methods out of `IOutputSender` when touching that seam. |
 | [#524](https://github.com/iamfatness/CoreVideoPro/issues/524), [#525](https://github.com/iamfatness/CoreVideoPro/issues/525) | GPU encode writer isolation and later recording/ISO/macOS adapter slices; separate from current edge I/O. |
@@ -190,6 +188,8 @@ These rows record specific fixes, not blanket production or fleet reliability.
 
 | Issue | Merged fix / acceptance |
 |---|---|
+| [#612](https://github.com/iamfatness/CoreVideoPro/issues/612) | The supervisor stand-off for an adapter's own retry backoff is fresh and bounded (one tallest rung plus start grace); a stale code escalates with a named reason. |
+| [#703](https://github.com/iamfatness/CoreVideoPro/issues/703) | Closed by owner disposition 2026-10-02: the YouTube audio lag is resolved. |
 | [#602](https://github.com/iamfatness/CoreVideoPro/issues/602) | A supervisor creates records only for the destinations its own sender serves; restart log lines name the supervisor. |
 | [#606](https://github.com/iamfatness/CoreVideoPro/issues/606), [#605](https://github.com/iamfatness/CoreVideoPro/issues/605) | RTMP destinations keep their own codec/fps/bitrate; the stream profile only fills unset fields. #605 (hardware GOP + IDR on demand) had landed in #707. |
 | [#762](https://github.com/iamfatness/CoreVideoPro/issues/762), [#741](https://github.com/iamfatness/CoreVideoPro/issues/741) | Core profile gates Engine/Record/Join in the shell; the core refuses Record on a failed encoder and Join with no engine; breakout simulator stub-only. Not yet run installed. |

native/src/modules/OutputDestinationSupervisorPolicy.h

@@ -209,6 +209,11 @@ class OutputDestinationSupervisorPolicy {
   static constexpr std::int64_t kProgressStaleMs = 1000;
   static constexpr std::int64_t kStalledRestartMs = 5000;
   static constexpr std::int64_t kStartGraceMs = 15000;
+  // #612: how long the supervisor will stand off for an adapter's self-managed
+  // retry before treating the latched code as stale. The adapter's own ladder
+  // tops out at kMaxBackoffMs per rung, so one full rung plus the start grace
+  // is the longest a live retry can legitimately go without a fresh result.
+  static constexpr std::int64_t kSelfManagedRetryStandoffMaxMs = kMaxBackoffMs + kStartGraceMs;
 
   // 5s, 10s, 20s, 40s, capped at 60s — identical to the four existing ladders.
   [[nodiscard]] static std::int64_t backoffMsForFailureCount(int consecutiveFailures) {
@@ -250,6 +255,7 @@ class OutputDestinationSupervisorPolicy {
     lastProgressMs_ = 0;
     faultPending_ = false;
     gaveUpAnnounced_ = gaveUp_;
+    standoffSinceMs_ = -1;
   }
 
   // Operator intervention: recover / re-select / re-arm. Always recoverable
@@ -339,15 +345,41 @@ class OutputDestinationSupervisorPolicy {
     // 5 s and the `never accepted output` branch after 15 s - the same walk to
     // give-up by a different door. While the adapter's own bounded retry is
     // pending, NO fault is armed at all.
-    const bool servingSelfManagedRetry = isSelfManagedRetryResultCode(o.lastResultCode);
+    // #612: the stand-off is keyed on a result code the adapter LATCHES, and the
+    // supervisor polls far more often than the adapter refreshes it. Two holes:
+    // give-up for a genuinely dead destination depended on which value the
+    // observer happened to see, and if frames stopped reaching the sender
+    // mid-rung the latched code was never refreshed, so nothing ended the
+    // stand-off. So the stand-off is now (a) FRESH: it needs this cycle's
+    // record, not a remembered one; and (b) BOUNDED: it cannot outlive the
+    // longest rung the adapter's own ladder can serve. Either failing hands the
+    // observation back to the ordinary fault rules, which escalate.
+    bool servingSelfManagedRetry = o.observed && isSelfManagedRetryResultCode(o.lastResultCode);
+    if (servingSelfManagedRetry) {
+      if (standoffSinceMs_ < 0) standoffSinceMs_ = o.nowMs;
+      // Progress ends a stand-off: the adapter is through its rung.
+      if (sinceProgressMs >= 0 && sinceProgressMs <= kProgressStaleMs) standoffSinceMs_ = o.nowMs;
+      if (o.nowMs - standoffSinceMs_ > kSelfManagedRetryStandoffMaxMs) {
+        servingSelfManagedRetry = false;
+        if (decision.failureClass == DestinationFailureClass::None) {
+          decision.failureClass = DestinationFailureClass::Retryable;
+        }
+      }
+    } else if (!isSelfManagedRetryResultCode(o.lastResultCode)) {
+      standoffSinceMs_ = -1;
+    }
     if (servingSelfManagedRetry && !faultPending_) {
       // Say so, so a support bundle shows standing off rather than silence.
       decision.reason = "Destination is serving its own bounded restart backoff (" +
                         o.lastResultCode + "); the supervisor is standing off.";
     }
     if (!faultPending_ && !servingSelfManagedRetry) {
       std::string faultReason;
-      if (decision.failureClass == DestinationFailureClass::Retryable) {
+      if (standoffSinceMs_ >= 0 && isSelfManagedRetryResultCode(o.lastResultCode)) {
+        faultReason = "Destination reported its own retry backoff (" + o.lastResultCode + ") for " +
+                      std::to_string((o.nowMs - standoffSinceMs_) / 1000) +
+                      "s without producing; the stand-off has expired.";
+      } else if (decision.failureClass == DestinationFailureClass::Retryable) {
         faultReason = o.lastError.empty()
                           ? ("Destination reported a failure (" + o.lastResultCode + ").")
                           : o.lastError;
@@ -427,6 +459,8 @@ class OutputDestinationSupervisorPolicy {
   std::int64_t lastProgressMs_ = 0;
   std::int64_t generationStartedMs_ = 0;
   std::int64_t nextAttemptAtMs_ = 0;
+  // When the current stand-off for a self-managed retry began; -1 when none.
+  std::int64_t standoffSinceMs_ = -1;
   std::string pendingReason_;
   std::string terminalReason_;
 };

native/tests/OutputDestinationSupervisorTest.cpp

@@ -489,8 +489,9 @@ TEST(OutputDestinationSupervisorPolicy, ADestinationServingItsOwnRestartBackoffI
 
   std::int64_t now = 100;
   int restarts = 0;
-  // Ten minutes of solid backoff: far past 5 x the tallest 60 s rung.
-  for (int tick = 0; tick < 2400; ++tick) {
+  // One full tallest rung of backoff (60 s) plus most of the grace: inside the
+  // #612 bound, so the stand-off holds for the whole of it.
+  for (int tick = 0; tick < 280; ++tick) {
     now += 250;  // the supervisor's own observation cadence
     const auto decision = policy.observe(activeObservation(now, 5, "failed", "ffmpeg-retry-backoff"));
     if (decision.action == SupervisorAction::Restart) {
@@ -514,9 +515,72 @@ TEST(OutputDestinationSupervisorPolicy, ADestinationServingItsOwnRestartBackoffI
   now += 250;
   const auto real = policy.observe(activeObservation(now, 5, "failed", "ffmpeg-exited"));
   EXPECT_EQ(real.failureClass, DestinationFailureClass::Retryable);
+  policy.onGenerationStarted(now);
+
+  // #612: AND THE STAND-OFF IS BOUNDED. A latched "ffmpeg-retry-backoff" that
+  // outlives the tallest rung the adapter can serve is stale, not a live retry
+  // (the shape when frames stop reaching the sender mid-rung). It must escalate
+  // through the ordinary ladder rather than hold the supervisor off forever.
+  int faults = 0;
+  std::string reason;
+  for (int tick = 0; tick < 2400 && faults == 0; ++tick) {
+    now += 250;
+    const auto decision = policy.observe(activeObservation(now, 5, "failed", "ffmpeg-retry-backoff"));
+    if (decision.failureClass != DestinationFailureClass::None) {
+      ++faults;
+      reason = decision.reason;
+    }
+  }
+  const std::int64_t standoffMs = now - 100 - 280 * 250 - 250;
+  EXPECT_EQ(faults, 1) << "a stand-off that never refreshed must expire and arm a fault";
+  EXPECT_GT(standoffMs, OutputDestinationSupervisorPolicy::kSelfManagedRetryStandoffMaxMs - 500);
+  // Two observation ticks of slack: the stand-off clock starts on the first
+  // backoff observation and the expiry fires on the tick after the bound.
+  EXPECT_LT(standoffMs, OutputDestinationSupervisorPolicy::kSelfManagedRetryStandoffMaxMs + 2000);
   EXPECT_FALSE(real.reason.empty());
 }
 
+// #612: the stand-off needs THIS cycle's record. A remembered code from a child
+// that has stopped publishing is not evidence that it is still retrying.
+TEST(OutputDestinationSupervisorPolicy, AnUnobservedBackoffCodeDoesNotHoldTheSupervisorOff) {
+  OutputDestinationSupervisorPolicy policy;
+  policy.onGenerationStarted(0);
+  policy.observe(activeObservation(100, 5, "live", "encoder-input-accepted"));
+  std::int64_t now = 100;
+  bool faulted = false;
+  for (int tick = 0; tick < 200 && !faulted; ++tick) {
+    now += 250;
+    auto observation = activeObservation(now, 5, "failed", "ffmpeg-retry-backoff");
+    observation.observed = false;  // no record arrived this cycle; the code is a memory
+    const auto decision = policy.observe(observation);
+    faulted = decision.failureClass != DestinationFailureClass::None || policy.faultPending();
+  }
+  EXPECT_TRUE(faulted) << "an unobserved destination that stopped accepting output must arm a fault";
+}
+
+TEST(OutputDestinationSupervisorPolicy, ProgressEndsAStandoffBeforeItCanExpire) {
+  OutputDestinationSupervisorPolicy policy;
+  policy.onGenerationStarted(0);
+  std::int64_t now = 0;
+  std::int64_t units = 0;
+  // Six minutes alternating 50 s of backoff with a burst of accepted output: a
+  // link that keeps coming back. The bound is per stand-off, so it never trips.
+  for (int cycle = 0; cycle < 6; ++cycle) {
+    for (int tick = 0; tick < 200; ++tick) {
+      now += 250;
+      const auto decision = policy.observe(activeObservation(now, units, "failed", "ffmpeg-retry-backoff"));
+      ASSERT_EQ(decision.failureClass, DestinationFailureClass::None) << "cycle " << cycle << " tick " << tick;
+    }
+    for (int tick = 0; tick < 8; ++tick) {
+      now += 250;
+      units += 10;
+      policy.observe(activeObservation(now, units, "live", "encoder-input-accepted"));
+    }
+  }
+  EXPECT_FALSE(policy.gaveUp());
+  EXPECT_EQ(policy.consecutiveFailures(), 0);
+}
+
 TEST(OutputDestinationSupervisorPolicy, AnUnknownFailureCodeIsRetryableNotTerminal) {
   // Guessing "terminal" would silently stop protecting a destination.
   EXPECT_FALSE(isTerminalResultCode("ffmpeg-exited"));