Describe the bug
When using the test-automation tracked sessions (TrackActivity().IncludeExternalTransports()) against a real RabbitMQ broker, tests intermittently fail with a NullReferenceException thrown from inside Wolverine's tracking bookkeeping, surfaced by TrackedSession.AssertNoExceptionsWereThrown():
System.AggregateException : One or more errors occurred. (Object reference not set to an instance of an object.)
---- System.NullReferenceException : Object reference not set to an instance of an object.
----- Inner Stack Trace -----
at System.Linq.Enumerable.TryGetLast[TSource](IEnumerable`1 source, Func`2 predicate, Boolean& found)
at Wolverine.Tracking.EnvelopeHistory.RecordCrossApplication(EnvelopeRecord record) in src/Wolverine/Tracking/EnvelopeHistory.cs:line 147
at Wolverine.Runtime.WolverineRuntime.Received(Envelope envelope) in src/Wolverine/Runtime/WolverineRuntime.Tracking.cs:line 86
at Wolverine.Runtime.HandlerPipeline.TryDeserializeEnvelope(Envelope envelope) in src/Wolverine/Runtime/HandlerPipeline.cs:line 189
at Wolverine.Runtime.HandlerPipeline.executeAsync(MessageContext context, Envelope envelope, Activity activity) in src/Wolverine/Runtime/HandlerPipeline.cs:line 218
at Wolverine.Runtime.HandlerPipeline.InvokeAsync(Envelope envelope, IChannelCallback channel, Activity activity) in src/Wolverine/Runtime/HandlerPipeline.cs:line 87
The failure is random (roughly 1 in 4 runs of a ~50-test integration suite in our case) and attaches to whatever test happens to have an active tracked session at that moment. The application message itself is unaffected — it is retried and handled normally; only the test fails.
Root cause (from reading the source)
EnvelopeHistory stores its records in a plain List<EnvelopeRecord> with no synchronization (still the case on current main):
private readonly List<EnvelopeRecord> _records = new();
RecordCrossApplication both iterates the list (LastOrDefault(...) → Enumerable.TryGetLast) and mutates it (_records.Add(record), index-based loops setting IsComplete). With IncludeExternalTransports(), WolverineRuntime.Received(...) is invoked concurrently from multiple transport listener threads (we have ~90 RabbitMQ queues listening), so two threads can be inside the same session's EnvelopeHistory at once — TryGetLast walks the list while another thread appends, producing a torn read and the NRE above.
Environment
- WolverineFx 6.4.1 (
WolverineFx.RabbitMQ, WolverineFx.SqlServer, WolverineFx.Http, WolverineFx.EntityFrameworkCore)
- .NET 10, xUnit v3 integration suite using
WebApplicationFactory
- RabbitMQ 4.2, conventional routing, durable inbox/outbox, ~90 queues on one host
To reproduce
Hard to reproduce deterministically since it's a data race. Conditions that make it likely:
- A host with many external (RabbitMQ) listeners and
IncludeExternalTransports() tracked sessions.
- Messages still arriving from a previous test/session while a new tracked session is active (the cross-application recording path).
- Run a suite of ~50+ tracked-session tests repeatedly; we hit it roughly every fourth run.
Suggested fix
A simple lock around the reads and writes of _records in EnvelopeHistory (it's test-support code, so contention cost is irrelevant), or replacing the list with a synchronized structure.
Workaround we use meanwhile
DoNotAssertOnExceptionsDetected() on the tracked session, then re-implementing the exception assertion from session.AllRecordsInOrder() while filtering out NullReferenceExceptions whose stack trace contains Wolverine.Tracking.
Describe the bug
When using the test-automation tracked sessions (
TrackActivity().IncludeExternalTransports()) against a real RabbitMQ broker, tests intermittently fail with aNullReferenceExceptionthrown from inside Wolverine's tracking bookkeeping, surfaced byTrackedSession.AssertNoExceptionsWereThrown():The failure is random (roughly 1 in 4 runs of a ~50-test integration suite in our case) and attaches to whatever test happens to have an active tracked session at that moment. The application message itself is unaffected — it is retried and handled normally; only the test fails.
Root cause (from reading the source)
EnvelopeHistorystores its records in a plainList<EnvelopeRecord>with no synchronization (still the case on currentmain):RecordCrossApplicationboth iterates the list (LastOrDefault(...)→Enumerable.TryGetLast) and mutates it (_records.Add(record), index-based loops settingIsComplete). WithIncludeExternalTransports(),WolverineRuntime.Received(...)is invoked concurrently from multiple transport listener threads (we have ~90 RabbitMQ queues listening), so two threads can be inside the same session'sEnvelopeHistoryat once —TryGetLastwalks the list while another thread appends, producing a torn read and the NRE above.Environment
WolverineFx.RabbitMQ,WolverineFx.SqlServer,WolverineFx.Http,WolverineFx.EntityFrameworkCore)WebApplicationFactoryTo reproduce
Hard to reproduce deterministically since it's a data race. Conditions that make it likely:
IncludeExternalTransports()tracked sessions.Suggested fix
A simple
lockaround the reads and writes of_recordsinEnvelopeHistory(it's test-support code, so contention cost is irrelevant), or replacing the list with a synchronized structure.Workaround we use meanwhile
DoNotAssertOnExceptionsDetected()on the tracked session, then re-implementing the exception assertion fromsession.AllRecordsInOrder()while filtering outNullReferenceExceptions whose stack trace containsWolverine.Tracking.