Package
Sentry.OpenTelemetry
.NET Flavor / Version
.NET 10.0.7 (also reproduced on .NET 8 and 9)
OS
Linux (production). Reproduces on any OS.
SDK Version
6.6.0
Summary
SentrySpanProcessor leaks spans whose Activity never receives OnEnd, for example aborted HTTP/2 streams or abandoned gRPC server streams. Such spans accumulate in SentrySpanProcessor._map and are never removed, so a long running service grows in memory until it runs out.
This is a sibling of #3197 / #3198. #3198 fixed the filtered span case, and its description mentions holding weak references to Activities so PruneFilteredSpans could also evict collected ones. But the reference is stored via SetFused, which is backed by a ConditionalWeakTable and holds its values strongly (only keys are weak). So _map -> span -> Activity pins the Activity alive, the activity is null branch in PruneFilteredSpans can never run, and recorded spans that never end stay in the map forever.
UseSentry / SentrySdk.Init call
builder.WebHost.UseSentry(o =>
{
o.Dsn = "<redacted>";
o.UseOpenTelemetry();
o.DisableSentryHttpMessageHandler = true;
o.TracesSampler = ctx => 0.1; // sample 10% of root traces
});
builder.Services.AddOpenTelemetry().WithTracing(t => t
.AddAspNetCoreInstrumentation()
.AddSentry());Steps to Reproduce
Minimal repro: https://gist.github.com/Ermabo/41e0301e0afabfe4d28cff799d4e79b6
dotnet run -c Release -- --close true (normal spans): memory stays flat.dotnet run -c Release -- --close false (Activities started but never stopped, i.e. an aborted request): memory grows without bound, about 1.9 GB over 300k iterations.
Expected Result
Spans whose Activity never ends are eventually evicted from _map, and memory stays bounded.
Actual Result
They are retained forever. In our production gRPC service we observed about 91k Sentry.Internal.UnsampledTransaction retained with _isFinished == false, gcroot terminating at SentrySpanProcessor._map. Managed gen2 grew from about 190 MB to 2.4 GB over roughly 6 days before an out of memory crash, repeating roughly every 4 days.
A fix PR is on the way. cc @jamescrosswell (author of #3198).
Package
Sentry.OpenTelemetry
.NET Flavor / Version
.NET 10.0.7 (also reproduced on .NET 8 and 9)
OS
Linux (production). Reproduces on any OS.
SDK Version
6.6.0
Summary
SentrySpanProcessorleaks spans whose Activity never receivesOnEnd, for example aborted HTTP/2 streams or abandoned gRPC server streams. Such spans accumulate inSentrySpanProcessor._mapand are never removed, so a long running service grows in memory until it runs out.This is a sibling of #3197 / #3198. #3198 fixed the filtered span case, and its description mentions holding weak references to Activities so
PruneFilteredSpanscould also evict collected ones. But the reference is stored viaSetFused, which is backed by aConditionalWeakTableand holds its values strongly (only keys are weak). So_map -> span -> Activitypins the Activity alive, theactivity is nullbranch inPruneFilteredSpanscan never run, and recorded spans that never end stay in the map forever.UseSentry / SentrySdk.Init call
Steps to Reproduce
Minimal repro: https://gist.github.com/Ermabo/41e0301e0afabfe4d28cff799d4e79b6
dotnet run -c Release -- --close true(normal spans): memory stays flat.dotnet run -c Release -- --close false(Activities started but never stopped, i.e. an aborted request): memory grows without bound, about 1.9 GB over 300k iterations.Expected Result
Spans whose Activity never ends are eventually evicted from
_map, and memory stays bounded.Actual Result
They are retained forever. In our production gRPC service we observed about 91k
Sentry.Internal.UnsampledTransactionretained with_isFinished == false,gcrootterminating atSentrySpanProcessor._map. Managed gen2 grew from about 190 MB to 2.4 GB over roughly 6 days before an out of memory crash, repeating roughly every 4 days.A fix PR is on the way. cc @jamescrosswell (author of #3198).