Composite projection does not catch up under the Wolverine-hosted daemon with UseTenantPartitionedEvents (Marten's own daemon does) #3083

Activity

1. erdtsieck commented on Jun 12, 2026

   

   erdtsieck

   on Jun 12, 2026

   ContributorAuthor

   More actions

   Narrowing — the JasperFx daemon is NOT at fault; it is specifically the Wolverine hosting layer:

   I added a second Marten-only repro that starts each shard via daemon.StartAgentAsync(shardName) (the exact method EventSubscriptionAgent.StartAsync calls) instead of StartAllAsync(), same config (Conjoined + QuickWithServerTimestamps + UseTenantPartitionedEvents + AllDocumentsAreMultiTenanted, 2 managed tenants, plain + composite, seed AFTER the daemon starts):

   • StartAllAsync() → composite catches up ✅
   • StartAgentAsync("CsComposite") + StartAgentAsync("CsPlain") → composite also catches up ✅

   So the Marten daemon drives the composite to completion via BOTH entry points, including the per-shard StartAgentAsync that Wolverine uses. The failure only appears when the same daemon is hosted by Wolverine's managed event-subscription distribution.

   Observed under Wolverine: the composite logs Starting single-pass composite rebuild for CsComposite:All over events 0 to 1 → Finished ... proceeding to normal, continuous operation, then never advances (stuck at 1) while the plain projection and the HighWaterMark both reach the real max(seq_id)=30. No auto-restart / stall messages are logged.

   So the defect is in the Wolverine event-subscription-agent hosting layer (EventSubscriptionAgentFamily / EventStoreAgents / EventSubscriptionAgent — managed-distribution path), in how a composite shard's continuous catch-up is driven after its initial (empty) optimized rebuild — not in JasperFxAsyncDaemon.StartAgentAsync itself. Reproduced with UseWolverineManagedEventSubscriptionDistribution = true; also reproduced with a plain IntegrateWithWolverine() (no managed flag). Repro tests available.
2. erdtsieck commented on Jun 12, 2026

   

   erdtsieck

   on Jun 12, 2026

   ContributorAuthor

   More actions

   Resolved — root cause is a stale Marten dependency, not a Wolverine-layer defect

   After reproducing this end-to-end I can confirm there is no bug in Wolverine's event-subscription distribution. The composite stall is the symptom of Marten #4712 (high-water Timestamp under UseTenantPartitionedEvents), and this repo's Directory.Packages.props pins Marten 9.6.0, which predates that fix.

   What actually happens on Marten 9.6.0

   With UseTenantPartitionedEvents, the store-global HighWaterStatisticsDetector read its Timestamp off the (absent) store-global mt_event_progression row, leaving it at default(DateTimeOffset) = 0001-01-01. The HighWaterAgent then computes safeHarborTime = _current.Timestamp + StaleSequenceThreshold = 0001-01-01 00:00:03, can never determine a safe harbor, and stops advancing:

   Starting single-pass composite rebuild for CsComposite:All over events 0 to 1
   Shard 'CsPlain:All': Executed updates for Event range ... 0 to 30
   Finished ... CsComposite:All, proceeding to normal, continuous operation
   High Water agent is stale at 30 ... skipping gap to events marked after 01/01/0001 00:00:03
   Daemon projection high water detection was not able to determine a safe harbor sequence, will try again soon   (forever)
   

   The composite finishes its premature 0 → 1 optimized rebuild, transitions to continuous, and is then never told to advance to the real high-water (30). The plain projection caught up only because it happened to consume the first good mark before the agent wedged. That ordering-sensitivity is also why this looked flaky.

   Proof (clean A/B, nothing else changed)

   The repro test (composite_side_effects_under_managed_distribution, the no-side-effects mirror of Marten's own passing repro) behaves like this:

   Marten	Result
   9.6.0 (this repo's pin)	hang ~90s, composite stuck at 1, stale … 01/01/0001
   9.7.3	PASS in ~4s

   Marten 9.7.3 already contains #4712. The Marten-only repros passed all along precisely because they build from 9.7.3 source.

   Conclusion

   No Wolverine code change is required — the daemon distribution layer is correct. The fix is to consume Marten ≥ 9.7.3 (which also pulls JasperFx 2.9.9). Closing as resolved-by-dependency. Bumping the pinned Marten version can be tracked separately if desired.

   Validation: bumping this repo's pinned Marten 9.6.0 → 9.7.3 (+ JasperFx 2.8.2 → 2.9.9) builds the full wolverine.slnx clean (0 warnings / 0 errors) and turns the repro green.
3. erdtsieck

   closed this as not plannedon Jun 12, 2026