#725 Accept a respawned core's roster instead of keeping the dead core's

CLAUDE.md

@@ -848,12 +848,30 @@ comment at the code site; this is the index.
   - **What happened:** in a long live session, every Take resolved Show Input slots against a
     participant list missing the producer (the app's own camera-off user) for one apply. The
     multiview layout lost a tile and the wall reflowed 5+4 to 4+4 and back on every cut. It
-    did not reproduce after a restart, and the source of the short list is still unknown.
-  - **What changed:** `MultiviewParticipantGrace` keeps an in-show slot's participant on the
-    layout for 1 s through such a dip. A real leave is still dropped after the grace.
-  - **Log line:** every miss logs `mv-roster-miss: slotN pid=... missing from <caller> list ...
-    coreRosterHasIt=yes|no` regardless of verbose. If it recurs, that line says whether the
-    shell or the core produced the short list.
+    did not reproduce after an app restart.
+  - **Root cause (found 2026-10-01): a core the supervisor RESPAWNED is a new roster authority,
+    and the shell rejected it.** The roster epoch is `process:meeting:token`. A new core process
+    starts again at `1:1`, and only the diagnostic token differs, so
+    `ZoomRosterSnapshotPolicy.Accept` could not order it after the dead core's barrier. That
+    policy relies on "a core restart clears the bridge snapshot". `StopCore` did; the
+    supervisor's own crash respawn did not. Consequences, for the rest of the app session:
+    - The bridge snapshot kept the DEAD core's roster. Joins, leaves and the producer's new
+      Zoom id never reached the shell, and slots kept pointing at ids no longer in the meeting.
+    - A Take applies its own raw sync response, which carried the true roster, and the next
+      bridge snapshot put the stale one back. That alternation was the dip.
+    - Measured live: bridge `1:1:18772-0 rev 38 n=8` against core `1:1:23540-0 rev 11 n=9`.
+  - **Fix:** `MediaCoreBridgeService.ReleaseRosterBarrierForRespawnedCore` drops the barrier
+    (epoch and revision only, participants kept) at the respawned core's handshake, keyed on the
+    supervisor restart count. Test: `RosterAfterCoreRespawnTests` (a fake core that dies and
+    comes back on a same-numbered epoch at a lower revision).
+  - **Still in place:** `MultiviewParticipantGrace` holds an in-show slot's participant for 1 s
+    through a dip; a real leave is dropped after the grace.
+  - **Log lines:** `mv-roster-miss: slotN pid=... missing from <caller> list ...
+    coreRosterHasIt=yes|no` (not for an empty pre-join roster), and `roster-drop: pid=...
+    origin=<apply path> ... snap(epoch= rev=) bridge(epoch= rev=)` whenever someone leaves the
+    shell's roster. Two different epochs on that line is this defect; one epoch is a real leave.
+  - **Rule:** "it only happens in a long session" usually means "after an event the session
+    eventually has". Read `media-core.log` for a respawn before theorising about accumulation.
 - **AN EMPTY RENDER PLAN IS NOT "DRAW NOTHING" (2026-08-15, CoreVideo Tiles T1).**
   All THREE compositors — `D3D11CompositorAdapter::resolveLayers`,
   `ProgramFramePreview`'s `buildProgramFramePreview`, and

docs/BACKLOG.md

@@ -58,6 +58,8 @@ These can preempt Next on a show night. They do not jump the ranked Now item unt
 
 | Issue | Remaining work |
 |---|---|
+| [#728](https://github.com/iamfatness/CoreVideoPro/issues/728) | The core died three times in 2.5 minutes on `std::bad_alloc` in the FFmpeg media decoder's frame allocation while Windows was low on virtual memory (three dumps, one stack). The decoder must drop the frame and stay up; audit other worker threads for the same exposure. Owner has not assigned a rank. |
+| [#725](https://github.com/iamfatness/CoreVideoPro/issues/725) | Root cause fixed in #727: the shell rejected a respawned core's roster. Owner acceptance on an installed beta remains. |
 | [#705](https://github.com/iamfatness/CoreVideoPro/issues/705) | Headless media A/V pattern reaches Program video but not audio. The media transport demand clock mismatches the render timeline; fix and verify PCM in a real Release recording. Owner has not assigned a rank. |
 | [#668](https://github.com/iamfatness/CoreVideoPro/issues/668) | Slices A–D merged (#670–#673): unique guest bindings, host exclusion, talk ledger and Set & Forget holds. Installed host-plus-three-guests, interview, solo and share-priority acceptance remains. |
 | [#674](https://github.com/iamfatness/CoreVideoPro/issues/674) | Slices A–F merged (#675–#680), including direct negotiated-format facts and existing Multiview texture crop. Installed no-click video-off, ISO arm, Engine-off transitions and ten-row GPU load remain acceptance; reopened after an early close. |

native-shell/CoreVideoPro.MediaCore.Tests/RosterAfterCoreRespawnTests.cs

@@ -0,0 +1,95 @@
+using CoreVideoPro.MediaCore.Models;
+using CoreVideoPro.MediaCore.Services;
+using Xunit;
+
+namespace CoreVideoPro.MediaCore.Tests;
+
+/// <summary>
+/// #725: a core the supervisor respawned is a NEW roster authority. Its epoch restarts at the
+/// same process and meeting numbers as the dead one (only the diagnostic token differs), so it
+/// does not order after the installed barrier. Live 2026-10-01: the shell kept the dead core's
+/// roster for the rest of the session and every Take flipped the multiview between the two.
+/// </summary>
+public sealed class RosterAfterCoreRespawnTests
+{
+    [Fact]
+    public async Task ARespawnedCoresRosterReplacesTheDeadCoresRoster()
+    {
+        var directory = Path.Combine(Path.GetTempPath(), "corevideo-roster-respawn-" + Guid.NewGuid().ToString("N"));
+        Directory.CreateDirectory(directory);
+        try
+        {
+            var script = Path.Combine(directory, "core.cjs");
+            // The first process serves the old meeting roster at a HIGH revision and dies on
+            // command. Its replacement serves a different roster at a LOWER revision under an
+            // epoch with the same process and meeting numbers: the measured live shape
+            // (1:1:18772-0 rev 38, then 1:1:23540-0 rev 11).
+            await File.WriteAllTextAsync(script, """
+                const readline = require('node:readline');
+                const fs = require('node:fs');
+                const path = require('node:path');
+                const marker = path.join(__dirname, 'second-generation');
+                const second = fs.existsSync(marker);
+                fs.writeFileSync(marker, '1');
+                const profile = {name:'roster-respawn-fake',renderer:'software',maxProgramResolution:'1920x1080'};
+                const roster = second
+                  ? {meetingState:'in_meeting', rosterEpoch:'1:1:core-b', rosterRevision:11,
+                     participants:[{userId:'200',displayName:'Producer after',sourceGeneration:1,muted:false,videoOn:false}]}
+                  : {meetingState:'in_meeting', rosterEpoch:'1:1:core-a', rosterRevision:38,
+                     participants:[{userId:'100',displayName:'Producer before',sourceGeneration:1,muted:false,videoOn:false}]};
+                readline.createInterface({input:process.stdin}).on('line', line => {
+                  const m = JSON.parse(line);
+                  let response = {id:m.id,ok:true,type:m.type};
+                  if (m.type === 'handshake') response = {...response,protocolVersion:{major:1,minor:0},profile};
+                  if (m.type === 'media-core-sync') {
+                    if (m.commands?.some(c => c.type === 'test-die')) process.exit(9);
+                    response.state = {health:{frameCount:1},profile,meetingState:'in_meeting',programFrameCount:1};
+                  }
+                  if (m.type === 'zoom-snapshot') response.snapshot = roster;
+                  console.log(JSON.stringify(response));
+                });
+                """);
+            var supervisor = new MediaCoreSupervisor(new MediaCoreSupervisorOptions
+            {
+                Command = "node", Args = [script], WorkingDirectory = directory,
+                HandshakeRequestTimeoutMs = 5000, RequestTimeoutMs = 3000,
+                FrameDrainIntervalMs = 100000
+            });
+            await using var bridge = new MediaCoreBridgeService(supervisor);
+            Assert.NotNull(await bridge.StartAsync());
+            await bridge.GetZoomSnapshotAsync();
+            Assert.Equal("100", bridge.LastSnapshot!.Participants[0].UserId);
+
+            try { await bridge.SyncAsync([new NativeMediaCoreCommand { Type = "test-die" }]); }
+            catch (Exception) { /* the core died mid-request, as a crash does */ }
+            await UntilAsync(() => supervisor.Health.RestartCount >= 1 && !supervisor.Health.Recovering);
+
+            await UntilAsync(async () =>
+            {
+                try { await bridge.GetZoomSnapshotAsync(); return true; }
+                catch (Exception) { return false; }
+            });
+
+            Assert.Equal("1:1:core-b", bridge.LastSnapshot!.RosterEpoch);
+            Assert.Equal("200", Assert.Single(bridge.LastSnapshot.Participants).UserId);
+        }
+        finally
+        {
+            try { Directory.Delete(directory, recursive: true); }
+            catch (IOException) { }
+        }
+    }
+
+    private static Task UntilAsync(Func<bool> condition) => UntilAsync(() => Task.FromResult(condition()));
+
+    private static async Task UntilAsync(Func<Task<bool>> condition)
+    {
+        var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(20);
+        while (DateTime.UtcNow < deadline)
+        {
+            if (await condition()) return;
+            await Task.Delay(50);
+        }
+        Assert.True(await condition());
+    }
+}

native-shell/CoreVideoPro.MediaCore/Services/MediaCoreBridgeService.cs

@@ -50,7 +50,11 @@ public MediaCoreBridgeService(MediaCoreSupervisor? supervisor = null)
             HealthChanged?.Invoke(health);
         };
         _supervisor.StatusChanged += status => StatusChanged?.Invoke(status);
-        _supervisor.ProfileChanged += profile => ProfileChanged?.Invoke(profile);
+        _supervisor.ProfileChanged += profile =>
+        {
+            ReleaseRosterBarrierForRespawnedCore();
+            ProfileChanged?.Invoke(profile);
+        };
         _supervisor.ZoomRecovered += PublishCaptureSnapshot;
         _supervisor.ZoomRosterFactReceived += QueueRosterFact;
         _supervisor.ZoomVideoFrameReceived += frame => ZoomVideoFrameReceived?.Invoke(frame);
@@ -146,6 +150,33 @@ private void StopCore(Action stopSupervisor)
         foreach (var change in isoStopped) OutputLifecycleChanged?.Invoke(change);
     }
 
+    private int _rosterBarrierRestartCount;
+
+    /// <summary>
+    /// #725: a core the supervisor respawned is a new roster authority. Its epoch restarts at the
+    /// dead core's process and meeting numbers (only the diagnostic token differs), so
+    /// <see cref="ZoomRosterSnapshotPolicy"/> cannot order it after the installed barrier and
+    /// rejected its roster for the rest of the session. That policy relies on a core restart
+    /// clearing the barrier; <see cref="StopCore"/> did, the supervisor's own respawn did not.
+    /// Runs at the new core's handshake, before it has reported any roster. The participants are
+    /// kept until the rejoin replaces them, so slots are not freed during the gap.
+    /// </summary>
+    private void ReleaseRosterBarrierForRespawnedCore()
+    {
+        var restarts = _supervisor.Health.RestartCount;
+        lock (_gate)
+        {
+            if (restarts == _rosterBarrierRestartCount) return;
+            _rosterBarrierRestartCount = restarts;
+            if (_lastSnapshot is not null)
+                _lastSnapshot = _lastSnapshot with { RosterEpoch = null, RosterRevision = 0 };
+            _controlRecovery.ResetProcess();
+            _rosterRecoveryGeneration++;
+            _rosterRecoveryInFlight = false;
+            _rosterFacts.Reset();
+        }
+    }
+
     public void ConfigureZoomSpineSync(Func<CancellationToken, Task<Dictionary<string, object?>>>? payloadFactory)
     {
         lock (_gate) { _spinePayloadFactory = payloadFactory; _spineFactoryVersion++; }

native-shell/CoreVideoPro.WinUI.Tests/MultiviewParticipantGraceTests.cs

@@ -35,6 +35,18 @@ public void APresentRosterIsReturnedUnchangedAndLogsNothing()
         Assert.Empty(log);
     }
 
+    // Before a join the roster is empty and the persisted slots name nobody present. That is
+    // not a dip: every launch logged ten "dropped from the multiview" lines (2026-10-01).
+    [Fact]
+    public void AnEmptyRosterBeforeAMeetingLogsNothing()
+    {
+        var (slots, _) = Show();
+        var log = new List<string>();
+        var grace = new MultiviewParticipantGrace(() => 1000, log.Add);
+        Assert.Empty(grace.Resolve(slots, [], "production-sync", _ => false));
+        Assert.Empty(log);
+    }
+
     // The measured case: slot 9's participant (the producer) is missing from one apply.
     [Fact]
     public void AOneApplyDipKeepsTheWallAtNineTiles()

native-shell/CoreVideoPro.WinUI/Services/MultiviewParticipantGrace.cs

@@ -67,7 +67,8 @@ public IReadOnlyList<Participant> Resolve(
             var ageMs = seen ? now - last.SeenAtMs : -1;
             var holding = seen && ageMs <= (long)Grace.TotalMilliseconds;
             var key = $"{slot.SlotNumber}:{pid}:{(holding ? "hold" : "drop")}";
-            if (_reportedMisses.Add(key))
+            // An empty roster is "not in a meeting", not a dip: persisted slots name nobody yet.
+            if (current.Count > 0 && _reportedMisses.Add(key))
             {
                 var core = coreRosterHas is null ? "unknown" : coreRosterHas(pid) ? "yes" : "no";
                 _log($"mv-roster-miss: slot{slot.SlotNumber} pid={pid} missing from {caller} list " +

native-shell/CoreVideoPro.WinUI/ViewModels/StudioViewModel.cs

@@ -3641,7 +3641,7 @@
         }
     }
 
     partial void OnActiveSceneIdChanged(string oldValue, string value)
     {
         if (!_scenes.Any(scene => string.Equals(scene.Id, value, StringComparison.Ordinal)))
         {
@@ -9044,6 +9044,7 @@
             if (pending.SceneId == ActiveSceneId)
                 _lowerThirdFreshness.Acknowledge(pending.LowerThirdRevision, snapshot.ProgramFrame?.FrameNumber ?? 0);
             if (pending.Version != _productionSyncCaptureVersion || pending.SceneId != ActiveSceneId) return false;
+            NoteParticipantApply($"sync-response:{reason ?? "none"}", snapshot);
             ApplyLiveProductionPatch(LiveProductionSync.MapSnapshotToStudioPatch(snapshot, BuildLiveProductionContext()));
             CommandStatus = $"{pending.SceneName} synced to media core";
             return true;
@@ -10494,6 +10495,7 @@
 
         var patch = LiveProductionSync.MapSnapshotToStudioPatch(snapshot, liveProductionContext);
         Tm("mapPatch");
+        NoteParticipantApply("snapshot-changed:patch", snapshot);
         ApplyLiveProductionPatch(patch);
         Tm("applyPatch");
         ApplyGraphicsAndCaptionStateFromSnapshot(snapshot);
@@ -10692,6 +10694,41 @@
 
     private string _liveStructureSignature = "";
 
+    // #725: a participant leaving the shell's roster logs once, naming the apply path that
+    // delivered the shorter list and the roster epoch it read next to the bridge's. That pair is
+    // what found the cause: a sync response on the respawned core's epoch alternating with a
+    // bridge snapshot still on the dead core's. A real leave shows the same epoch on both.
+    private string _participantApplyOrigin = "unknown";
+    private NativeMediaCoreStateSnapshot? _participantApplySnapshot;
+
+    private void NoteParticipantApply(string origin, NativeMediaCoreStateSnapshot? snapshot)
+    {
+        _participantApplyOrigin = origin;
+        _participantApplySnapshot = snapshot;
+    }
+
+    private void LogRosterDrops(IReadOnlyList<Participant> mapped, IReadOnlyList<Participant> roomParticipants)
+    {
+        var previous = RoomParticipantsForInputs;
+        if (previous.Count == 0)
+        {
+            return;
+        }
+
+        var present = roomParticipants.Select(participant => participant.Id).ToHashSet(StringComparer.Ordinal);
+        var snapshot = _participantApplySnapshot;
+        var bridge = _bridge.LastSnapshot;
+        foreach (var gone in previous.Where(participant => !present.Contains(participant.Id)))
+        {
+            var inMapped = mapped.FirstOrDefault(participant => participant.Id == gone.Id);
+            LaunchLog.Write(
+                $"roster-drop: pid={gone.Id} origin={_participantApplyOrigin} prevN={previous.Count} newN={roomParticipants.Count} " +
+                $"mappedN={mapped.Count} mappedRoom={(inMapped is null ? "absent" : $"'{inMapped.BreakoutRoomId}'")} currentRoom='{_currentRoomId}' " +
+                $"snap(epoch={snapshot?.RosterEpoch ?? "null"} rev={snapshot?.RosterRevision ?? -1} n={snapshot?.Participants.Count ?? -1} state={snapshot?.MeetingState ?? "null"}) " +
+                $"bridge(epoch={bridge?.RosterEpoch ?? "null"} rev={bridge?.RosterRevision ?? -1} n={bridge?.Participants.Count ?? -1} same={ReferenceEquals(bridge, snapshot)})");
+        }
+    }
+
     private void ApplyLiveParticipants(IReadOnlyList<LiveProductionSync.LiveProductionParticipantContext> participants)
     {
         var mapped = ParticipantMapper.ToParticipants(participants);
@@ -10700,6 +10737,7 @@
         // expensive bound-list rebuild. Audio mixing and its Zoom mute indicator
         // read these room snapshots, including guests whose cameras are off.
         var roomParticipants = ParticipantMapper.ParticipantsInRoom(mapped, _currentRoomId);
+        LogRosterDrops(mapped, roomParticipants);
         var muteChanged = ParticipantMapper.HasMuteChanges(RoomParticipantsForInputs, roomParticipants);
         RoomParticipantsForInputs = roomParticipants;
         RoomVideoParticipants = ParticipantMapper.VideoParticipantsInRoom(mapped, _currentRoomId);
@@ -10920,6 +10958,7 @@
             Settings.ApplyMeetingStateLabel(meetingState, participants?.Count ?? snapshot.Participants.Count);
             if (participants is { } && (participants.Count > 0 || snapshot.RosterRevision > 0))
             {
+                NoteParticipantApply("meeting-fields", snapshot);
                 ApplyLiveParticipants(participants);
                 MfT("applyParticipants");
                 SyncShowInputsFromMeeting(participants);