No supported way to force async projections to catch up under Wolverine-managed event subscription distribution (marten#4904 left the active path to the coordinator) #3697

Under UseWolverineManagedEventSubscriptionDistribution there is no supported way to force async projections to catch up in an integration test, so every suite has to reimplement "quiesce the agents, then force catch-up" — and racing the still-running daemon while doing so.

Background

marten#4904 changed IHost.ForceAllMartenDaemonActivityToCatchUpAsync() to a passive read-only wait when the store runs in DaemonMode.ExternallyManaged, which is the mode Wolverine puts Marten in under Wolverine-managed event-subscription distribution. It no longer builds projections on demand, so a "caught up" wait can return before the read models exist, and the issue explicitly left the active path ("quiesce the external agents, then force catch-up") to the external coordinator. As far as we can tell Wolverine does not implement that path, so there is nothing to call.

What we ended up doing

Our integration suite (sharded multi-tenant store, UseTenantPartitionedEvents, agents distributed by Wolverine) drives it by hand:

var coordinator = services.GetRequiredService<IProjectionCoordinator>();
await coordinator.PauseAsync();               // stop agent assignment so CatchUp is the sole writer
try
{
    foreach (var daemon in await coordinator.AllDaemonsAsync())
    {
        await daemon.StopAllAsync();
        await daemon.CatchUpAsync(cancellation);
    }
}
finally
{
    await coordinator.StartAsync(cancellation);
}

It works, but "sole writer" is not actually guaranteed: the running daemon and the forced catch-up still overlap briefly, and per test run we see a handful of

• ProgressionProgressOutOfOrderException ("multiple processes try to process the projection"), and
• 23505: duplicate key value violates unique constraint "pk_mt_event_progression", Key (name)=(<projection>:V<n>:All:<tenant>) already exists — the same race on the first progression row of a shard, where the batch still INSERTs rather than UPDATEs.

Both are transient (a retried catch-up updates the row that now exists), so we retry a few times and then swallow exactly those two. Every suite that uses Wolverine-managed distribution presumably needs the same private workaround, including the knowledge of which exceptions are "just the race".

To be clear: we are not asking you to make the duplicate key go away in our workaround — we are asking for the supported path so the workaround is unnecessary.

Ask

Something like IProjectionCoordinator.ForceCatchUpAsync(CancellationToken) (or making Marten's ForceAllMartenDaemonActivityToCatchUpAsync delegate to the coordinator when the store is ExternallyManaged) that:

1. suspends agent assignment cluster-wide for the duration, so the forced catch-up really is the only writer of each progression row;
2. drives every locally reachable daemon to catch up, including per-(database, tenant) shards;
3. restores the previous assignment state afterwards.

Even without cluster-wide coordination, a single-node/test-oriented variant would remove the two failure modes above, because the insert/update race only exists while both writers are live.

Happy to test a prerelease against a suite that reproduces this a few times per run.

Versions

WolverineFx 6.24.0, Marten 9.20.2, JasperFx(.Events) 2.36.2, .NET 10, PostgreSQL 17.