Skip to content

Adopt Marten 9.14.0+: implement external-coordinator active catch-up for managed event-subscription distribution (Marten #4904) #3349

Description

@jeremydmiller

Blocker found while bumping deps (#3348)

Marten 9.13.0-stable and 9.14.0 hang Wolverine's daemon catch-up under
UseWolverineManagedEventSubscriptionDistribution. These MartenTests.TestHelpers tests time out:

  • catch_up_when_using_wolverine_distribution (main + ancillary store)
  • catch_up_then_restart
  • second_stage_waiting
  • catch_up_and_then_do_nothing

Only the currently-pinned 9.13.0-alpha.4 passes, so #3348 keeps Marten at alpha.4.

Root cause

Marten #4904 (shipped in 9.14.0) intentionally changed
ForceAllMartenDaemonActivityToCatchUpAsync so that when the store is
DaemonMode.ExternallyManaged (what Wolverine-managed distribution uses) it no longer drives
Stop→CatchUp→Start. Instead it degrades to a read-only WaitForNonStaleDataAsync wait, to avoid a
ProgressionProgressOutOfOrderException race where both Marten's forced catch-up and Wolverine's
externally-owned agent advance the same mt_event_progression row. Marten's own comment:

The proper active-quiesce — the external coordinator suspending agent assignment during a forced
catch-up — lives in that external coordinator, not here; tracked on #4904.

Wolverine's PauseThenCatchUpOnMartenDaemonActivity (src/Persistence/Wolverine.Marten/TestingExtensions.cs)
pauses the daemons and then relies on ForceAll to drive them forward. Under 9.14.0 the call just
waits; the paused agents never advance; it blocks until the tracked-session timeout.

What Wolverine needs

Implement the "external coordinator" active-quiesce catch-up that Marten #4904 punted to us — most
likely on WolverineProjectionCoordinator:

  1. Suspend Wolverine agent assignment/reassignment for the store's subscription agents.
  2. Force those agents (which Wolverine owns) to catch up to the current high-water mark.
  3. Resume normal assignment.

Then update PauseThenCatchUpOnMartenDaemonActivity (and any production catch-up path) to use that
coordinator-driven catch-up under managed distribution instead of Marten's now-passive ForceAll.

Once done, bump Marten to 9.14.0+ and re-run the catch-up suite.

Repro

Check out #3348 with Marten pinned to 9.14.0 (or 9.13.0 stable) and run:
dotnet test src/Persistence/MartenTests/MartenTests.csproj --framework net9.0 --filter "FullyQualifiedName~TestHelpers.catch_up_when_using_wolverine_distribution"
— fails (TimeoutException) on 9.14.0/9.13.0-stable, passes on 9.13.0-alpha.4.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions