Closed
Description
Description
I have encountered a deadlock on EventSource.EventListenersLock.
The thread holding the lock is in EventPipeEventDispatcher.StopDispatchTask() doing m_dispatchTask.Wait():
> System.Private.CoreLib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 575 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 3021 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.InternalWaitCore(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 2956 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 2752 C#
[Waiting on Async Operation, double-click or press enter to view Async Call Stacks]
System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventDispatcher.StopDispatchTask() Line 153 C#
System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventDispatcher.CommitDispatchConfiguration() Line 88 C#
System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventDispatcher.SendCommand(System.Diagnostics.Tracing.EventListener eventListener, System.Diagnostics.Tracing.EventCommand command, bool enable, System.Diagnostics.Tracing.EventLevel level, System.Diagnostics.Tracing.EventKeywords matchAnyKeywords) Line 57 C#
System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.EnableEvents(System.Diagnostics.Tracing.EventSource eventSource, System.Diagnostics.Tracing.EventLevel level, System.Diagnostics.Tracing.EventKeywords matchAnyKeyword, System.Collections.Generic.IDictionary<string, string> arguments) Line 4076 C#
prometheus-net.DotNetRuntime.dll!Prometheus.DotNetRuntime.EventListening.DotNetEventListener.OnEventSourceCreated(object sender, System.Diagnostics.Tracing.EventSourceCreatedEventArgs e) Unknown
System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.CallBackForExistingEventSources(bool addToListenersList, System.EventHandler<System.Diagnostics.Tracing.EventSourceCreatedEventArgs> callback) Line 4412 C#
System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.EventSourceCreated.add(System.EventHandler<System.Diagnostics.Tracing.EventSourceCreatedEventArgs> value) Line 3951 C#
prometheus-net.DotNetRuntime.dll!Prometheus.DotNetRuntime.EventListening.DotNetEventListener.DotNetEventListener(Prometheus.DotNetRuntime.EventListening.IEventListener eventListener, System.Diagnostics.Tracing.EventLevel level, Prometheus.DotNetRuntime.EventListening.DotNetEventListener.GlobalOptions globalOptions) Unknown
prometheus-net.DotNetRuntime.dll!Prometheus.DotNetRuntime.DotNetRuntimeStatsCollector.CreateEventListeners.AnonymousMethod__9_0(Prometheus.DotNetRuntime.ListenerRegistration r) Unknown
System.Linq.dll!System.Linq.Enumerable.SelectEnumerableIterator<Prometheus.DotNetRuntime.ListenerRegistration, Prometheus.DotNetRuntime.EventListening.DotNetEventListener>.ToArray() Unknown
prometheus-net.DotNetRuntime.dll!Prometheus.DotNetRuntime.DotNetRuntimeStatsCollector.CreateEventListeners() Unknown
prometheus-net.DotNetRuntime.dll!Prometheus.DotNetRuntime.DotNetRuntimeStatsCollector.DotNetRuntimeStatsCollector(Microsoft.Extensions.DependencyInjection.ServiceProvider serviceProvider, Prometheus.CollectorRegistry metricRegistry, Prometheus.DotNetRuntime.DotNetRuntimeStatsCollector.Options options) Unknown
prometheus-net.DotNetRuntime.dll!Prometheus.DotNetRuntime.DotNetRuntimeStatsBuilder.Builder.StartCollecting(Prometheus.CollectorRegistry registry) Unknown
[snip]
m_dispatchTask is meanwhile busy dispatching events and one of these dispatches has triggered code that wants to take the same lock:
Reproduction Steps
I do not have an easy way to consistently reproduce the issue but I have a process dump of the deadlocked state, if it helps.
Expected behavior
No deadlocks when doing EventSource related opterations (in this case when accessing Activity.Current, which appears to use EventSource internally, though the expectation is general for all EventSource related operations).
Actual behavior
Deadlock occurs.
Regression?
No response
Known Workarounds
No response
Configuration
Windows 11 x64.
Reproduced both with .NET 7 and .NET 8 preview 3.
Other information
I have a process memory dump, if it helps. Ping me on Teams (sasaares).