Skip to content

Intermittent NullReferenceException from EnvelopeHistory.RecordCrossApplication during tracked sessions — EnvelopeHistory is not thread-safe #3069

Description

@alesdvorakcz

Describe the bug

When using the test-automation tracked sessions (TrackActivity().IncludeExternalTransports()) against a real RabbitMQ broker, tests intermittently fail with a NullReferenceException thrown from inside Wolverine's tracking bookkeeping, surfaced by TrackedSession.AssertNoExceptionsWereThrown():

System.AggregateException : One or more errors occurred. (Object reference not set to an instance of an object.)
---- System.NullReferenceException : Object reference not set to an instance of an object.
----- Inner Stack Trace -----
   at System.Linq.Enumerable.TryGetLast[TSource](IEnumerable`1 source, Func`2 predicate, Boolean& found)
   at Wolverine.Tracking.EnvelopeHistory.RecordCrossApplication(EnvelopeRecord record) in src/Wolverine/Tracking/EnvelopeHistory.cs:line 147
   at Wolverine.Runtime.WolverineRuntime.Received(Envelope envelope) in src/Wolverine/Runtime/WolverineRuntime.Tracking.cs:line 86
   at Wolverine.Runtime.HandlerPipeline.TryDeserializeEnvelope(Envelope envelope) in src/Wolverine/Runtime/HandlerPipeline.cs:line 189
   at Wolverine.Runtime.HandlerPipeline.executeAsync(MessageContext context, Envelope envelope, Activity activity) in src/Wolverine/Runtime/HandlerPipeline.cs:line 218
   at Wolverine.Runtime.HandlerPipeline.InvokeAsync(Envelope envelope, IChannelCallback channel, Activity activity) in src/Wolverine/Runtime/HandlerPipeline.cs:line 87

The failure is random (roughly 1 in 4 runs of a ~50-test integration suite in our case) and attaches to whatever test happens to have an active tracked session at that moment. The application message itself is unaffected — it is retried and handled normally; only the test fails.

Root cause (from reading the source)

EnvelopeHistory stores its records in a plain List<EnvelopeRecord> with no synchronization (still the case on current main):

private readonly List<EnvelopeRecord> _records = new();

RecordCrossApplication both iterates the list (LastOrDefault(...) → Enumerable.TryGetLast) and mutates it (_records.Add(record), index-based loops setting IsComplete). With IncludeExternalTransports(), WolverineRuntime.Received(...) is invoked concurrently from multiple transport listener threads (we have ~90 RabbitMQ queues listening), so two threads can be inside the same session's EnvelopeHistory at once — TryGetLast walks the list while another thread appends, producing a torn read and the NRE above.

Environment

  • WolverineFx 6.4.1 (WolverineFx.RabbitMQ, WolverineFx.SqlServer, WolverineFx.Http, WolverineFx.EntityFrameworkCore)
  • .NET 10, xUnit v3 integration suite using WebApplicationFactory
  • RabbitMQ 4.2, conventional routing, durable inbox/outbox, ~90 queues on one host

To reproduce

Hard to reproduce deterministically since it's a data race. Conditions that make it likely:

  1. A host with many external (RabbitMQ) listeners and IncludeExternalTransports() tracked sessions.
  2. Messages still arriving from a previous test/session while a new tracked session is active (the cross-application recording path).
  3. Run a suite of ~50+ tracked-session tests repeatedly; we hit it roughly every fourth run.

Suggested fix

A simple lock around the reads and writes of _records in EnvelopeHistory (it's test-support code, so contention cost is irrelevant), or replacing the list with a synchronized structure.

Workaround we use meanwhile

DoNotAssertOnExceptionsDetected() on the tracked session, then re-implementing the exception assertion from session.AllRecordsInOrder() while filtering out NullReferenceExceptions whose stack trace contains Wolverine.Tracking.

Activity

  1. added a commit that references this issue on Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions