"Error starting gRPC call" - concurrency issue.

What version of gRPC and what language are you using?

C#

    <PackageReference Include="Grpc.AspNetCore" Version="2.45.0" />
    <PackageReference Include="Grpc.Net.ClientFactory" Version="2.45.0" />
    <PackageReference Include="Google.Protobuf" Version="3.20.1" />
    <PackageReference Include="Grpc" Version="2.45.0" />
    <PackageReference Include="Grpc.Tools" Version="2.45.0" PrivateAssets="All" />

What operating system (Linux, Windows,...) and version?

• Windows 10 (10.0.19044.0)
• Windows Server 2016 (10.0.14393.0)

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

Visual Studio 2022 17.1.3.

Windows 10 (10.0.19044.0) dotnet --info:

.NET SDK (reflecting any global.json):
 Version:   6.0.201
 Commit:    ef40e6aa06

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19044
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\6.0.201\

Host (useful for support):
  Version: 6.0.3
  Commit:  c24d9a9c91

.NET SDKs installed:
  1.0.0-preview2-003131 [C:\Program Files\dotnet\sdk]
  1.1.10 [C:\Program Files\dotnet\sdk]
  1.1.11 [C:\Program Files\dotnet\sdk]
  1.1.12 [C:\Program Files\dotnet\sdk]
  1.1.13 [C:\Program Files\dotnet\sdk]
  1.1.14 [C:\Program Files\dotnet\sdk]
  2.1.4 [C:\Program Files\dotnet\sdk]
  2.1.104 [C:\Program Files\dotnet\sdk]
  2.1.201 [C:\Program Files\dotnet\sdk]
  2.1.202 [C:\Program Files\dotnet\sdk]
  2.1.400 [C:\Program Files\dotnet\sdk]
  2.1.402 [C:\Program Files\dotnet\sdk]
  2.1.403 [C:\Program Files\dotnet\sdk]
  2.1.500 [C:\Program Files\dotnet\sdk]
  2.1.502 [C:\Program Files\dotnet\sdk]
  2.1.503 [C:\Program Files\dotnet\sdk]
  2.1.504 [C:\Program Files\dotnet\sdk]
  2.1.505 [C:\Program Files\dotnet\sdk]
  2.1.507 [C:\Program Files\dotnet\sdk]
  2.1.508 [C:\Program Files\dotnet\sdk]
  2.1.509 [C:\Program Files\dotnet\sdk]
  2.1.511 [C:\Program Files\dotnet\sdk]
  2.1.512 [C:\Program Files\dotnet\sdk]
  2.1.513 [C:\Program Files\dotnet\sdk]
  2.1.514 [C:\Program Files\dotnet\sdk]
  2.1.516 [C:\Program Files\dotnet\sdk]
  2.2.103 [C:\Program Files\dotnet\sdk]
  5.0.100 [C:\Program Files\dotnet\sdk]
  6.0.201 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.18 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 5.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.14 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.15 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download

Windows Server 2016 (10.0.14393.0) dotnet --info:

Host (useful for support):
  Version: 6.0.4
  Commit:  be98e88c76

.NET SDKs installed:
  No SDKs were found.

.NET runtimes installed:
  Microsoft.AspNetCore.App 5.0.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 5.0.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download

What did you do?

Hacked up csharp Helloworld / GreeterServer example to host a basic gRPC service that serves a simple unary HealthCheck() method via ASP.NET Core. The same process also implements custom HealthChecks to probe the aforementioned gRPC service and post results via HealthChecks-UI. See attached sample project, from which the logging below was generated.

What did you expect to see?

Expected to see the program execute without raising exceptions.

What did you see instead?

This exception, specifically:

fail: Grpc.Net.Client.Internal.GrpcCall[6]
      Error starting gRPC call.
      System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
         at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
         at System.Collections.Generic.Dictionary`2.set_Item(TKey key, TValue value)
         at Grpc.Net.Client.Configuration.ConfigObject.SetValue[T](String key, T value)
         at Grpc.Net.Client.Internal.Configuration.ConfigProperty`2.SetValue(ConfigObject inner, TValue value)
         at Grpc.Net.Client.Internal.Configuration.ConfigProperty`2.GetValue(ConfigObject inner)
         at Grpc.Net.Client.Configuration.ServiceConfig.get_LoadBalancingConfigs()
         at Grpc.Net.Client.Balancer.Internal.ConnectionManager.OnResolverResult(ResolverResult result)
         at Grpc.Net.Client.Balancer.StaticResolver.Start(Action`1 listener)
         at Grpc.Net.Client.Balancer.Internal.ConnectionManager.EnsureResolverStartedAsync()
         at Grpc.Net.Client.Balancer.Internal.ConnectionManager.ConnectAsync(Boolean waitForReady, CancellationToken cancellationToken)
         at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
         at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)

Complete set of logging the above exception was located within:

info: HealthChecks.UI.Core.HostedService.UIInitializationHostedService[0]
      Initializing UI Database
info: HealthChecks.UI.Core.HostedService.UIInitializationHostedService[0]
      Saving healthchecks configuration to database
dbug: HealthChecks.UI.Core.HostedService.HealthCheckCollectorHostedService[0]
      Executing HealthCheck collector HostedService.
dbug: Grpc.Net.Client.Internal.GrpcCall[1]
      Starting gRPC call. Method type: 'Unary', URI: 'http://localhost:30051/healthcheck.Greeter/HealthCheck'.
dbug: Grpc.Net.Client.Internal.GrpcCall[1]
      Starting gRPC call. Method type: 'Unary', URI: 'http://localhost:30051/healthcheck.Greeter/HealthCheck'.
dbug: Grpc.Net.Client.Internal.GrpcCall[1]
      Starting gRPC call. Method type: 'Unary', URI: 'http://localhost:30051/healthcheck.Greeter/HealthCheck'.
dbug: Grpc.Net.Client.Internal.GrpcCall[1]
      Starting gRPC call. Method type: 'Unary', URI: 'http://localhost:30051/healthcheck.Greeter/HealthCheck'.
dbug: Grpc.Net.Client.Balancer.Subchannel[1]
      Subchannel id '1' created with addresses: localhost:30051
dbug: Grpc.Net.Client.Balancer.Subchannel[1]
      Subchannel id '1' created with addresses: localhost:30051
dbug: Grpc.Net.Client.Balancer.Subchannel[1]
      Subchannel id '1' created with addresses: localhost:30051
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[4]
      Channel picker updated.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[4]
      Channel picker updated.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[4]
      Channel picker updated.
fail: Grpc.Net.Client.Internal.GrpcCall[6]
      Error starting gRPC call.
      System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
         at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
         at System.Collections.Generic.Dictionary`2.set_Item(TKey key, TValue value)
         at Grpc.Net.Client.Configuration.ConfigObject.SetValue[T](String key, T value)
         at Grpc.Net.Client.Internal.Configuration.ConfigProperty`2.SetValue(ConfigObject inner, TValue value)
         at Grpc.Net.Client.Internal.Configuration.ConfigProperty`2.GetValue(ConfigObject inner)
         at Grpc.Net.Client.Configuration.ServiceConfig.get_LoadBalancingConfigs()
         at Grpc.Net.Client.Balancer.Internal.ConnectionManager.OnResolverResult(ResolverResult result)
         at Grpc.Net.Client.Balancer.StaticResolver.Start(Action`1 listener)
         at Grpc.Net.Client.Balancer.Internal.ConnectionManager.EnsureResolverStartedAsync()
         at Grpc.Net.Client.Balancer.Internal.ConnectionManager.ConnectAsync(Boolean waitForReady, CancellationToken cancellationToken)
         at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
         at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
dbug: Grpc.Net.Client.Balancer.Subchannel[11]
      Subchannel id '1' state changed to Connecting. Detail: 'Connection requested.'.
dbug: Grpc.Net.Client.Balancer.Subchannel[11]
      Subchannel id '1' state changed to Connecting. Detail: 'Connection requested.'.
dbug: Grpc.Net.Client.Balancer.Subchannel[11]
      Subchannel id '1' state changed to Connecting. Detail: 'Connection requested.'.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[3]
      Channel state updated to Connecting.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[3]
      Channel state updated to Connecting.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[3]
      Channel state updated to Connecting.
info: Grpc.Net.Client.Internal.GrpcCall[3]
      Call failed with gRPC error status. Status code: 'Internal', Message: 'Error starting gRPC call. InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.'.
dbug: Grpc.Net.Client.Internal.GrpcCall[4]
      Finished gRPC call.
dbug: Grpc.Net.Client.Internal.GrpcCall[8]
      gRPC call canceled.
dbug: Grpc.Net.Client.Balancer.Internal.SocketConnectivitySubchannelTransport[2]
      Subchannel id '1' connected to socket localhost:30051.
dbug: Grpc.Net.Client.Balancer.Internal.SocketConnectivitySubchannelTransport[2]
      Subchannel id '1' connected to socket localhost:30051.
dbug: Grpc.Net.Client.Balancer.Internal.SocketConnectivitySubchannelTransport[2]
      Subchannel id '1' connected to socket localhost:30051.
dbug: Grpc.Net.Client.Balancer.Subchannel[11]
      Subchannel id '1' state changed to Ready. Detail: 'Successfully connected to socket.'.
dbug: Grpc.Net.Client.Balancer.Subchannel[11]
      Subchannel id '1' state changed to Ready. Detail: 'Successfully connected to socket.'.
dbug: Grpc.Net.Client.Balancer.Subchannel[11]
      Subchannel id '1' state changed to Ready. Detail: 'Successfully connected to socket.'.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[3]
      Channel state updated to Ready.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[3]
      Channel state updated to Ready.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[3]
      Channel state updated to Ready.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[4]
      Channel picker updated.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[4]
      Channel picker updated.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[4]
      Channel picker updated.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[6]
      Successfully picked subchannel id '1' with address localhost:30051.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[6]
      Successfully picked subchannel id '1' with address localhost:30051.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[6]
      Successfully picked subchannel id '1' with address localhost:30051.
warn: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check Greeter Service Health Check 1 with status Degraded completed after 45.1026ms with message 'HealthCheckRequest resulted in RpcException with StatusCode 'Internal''
dbug: Grpc.Net.Client.Internal.GrpcCall[18]
      Sending message.
dbug: Grpc.Net.Client.Internal.GrpcCall[18]
      Sending message.
dbug: Grpc.Net.Client.Internal.GrpcCall[18]
      Sending message.
dbug: Grpc.AspNetCore.Server.ServerCallHandler[10]
      Reading message.
dbug: Grpc.AspNetCore.Server.ServerCallHandler[10]
      Reading message.
dbug: Grpc.AspNetCore.Server.ServerCallHandler[10]
      Reading message.
dbug: Grpc.AspNetCore.Server.ServerCallHandler[15]
      Sending message.
dbug: Grpc.AspNetCore.Server.ServerCallHandler[15]
      Sending message.
dbug: Grpc.AspNetCore.Server.ServerCallHandler[15]
      Sending message.
dbug: Grpc.Net.Client.Internal.GrpcCall[8]
      gRPC call canceled.
dbug: Grpc.Net.Client.Internal.GrpcCall[8]
      gRPC call canceled.
dbug: Grpc.Net.Client.Internal.GrpcCall[8]
      gRPC call canceled.
dbug: Grpc.Net.Client.Internal.GrpcCall[13]
      Reading message.
dbug: Grpc.Net.Client.Internal.GrpcCall[13]
      Reading message.
dbug: Grpc.Net.Client.Internal.GrpcCall[13]
      Reading message.
info: Grpc.Net.Client.Internal.GrpcCall[3]
      Call failed with gRPC error status. Status code: 'Cancelled', Message: 'Call canceled by the client.'.
dbug: Grpc.Net.Client.Internal.GrpcCall[4]
      Finished gRPC call.
info: Grpc.Net.Client.Internal.GrpcCall[3]
      Call failed with gRPC error status. Status code: 'Cancelled', Message: 'Call canceled by the client.'.
dbug: Grpc.Net.Client.Internal.GrpcCall[4]
      Finished gRPC call.
info: Grpc.Net.Client.Internal.GrpcCall[3]
      Call failed with gRPC error status. Status code: 'Cancelled', Message: 'Call canceled by the client.'.
dbug: Grpc.Net.Client.Internal.GrpcCall[4]
      Finished gRPC call.
warn: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check Greeter Service Health Check 4 with status Degraded completed after 103.0864ms with message 'HealthCheckRequest resulted in RpcException with StatusCode 'Cancelled''
warn: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check Greeter Service Health Check 2 with status Degraded completed after 103.0864ms with message 'HealthCheckRequest resulted in RpcException with StatusCode 'Cancelled''
warn: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check Greeter Service Health Check 3 with status Degraded completed after 103.0873ms with message 'HealthCheckRequest resulted in RpcException with StatusCode 'Cancelled''
dbug: HealthChecks.UI.Core.HostedService.HealthCheckReportCollector[0]
      HealthReportCollector - health report execution history saved.
dbug: HealthChecks.UI.Core.HostedService.HealthCheckReportCollector[0]
      Creating a new HealthReport history.
dbug: HealthChecks.UI.Core.HostedService.HealthCheckReportCollector[0]
      HealthReportCollector has completed.
dbug: HealthChecks.UI.Core.HostedService.HealthCheckCollectorHostedService[0]
      HealthCheck collector HostedService executed successfully.
dbug: Grpc.Net.Client.Balancer.Subchannel[11]
      Subchannel id '1' state changed to Idle. Detail: 'Disconnected.'.
dbug: Grpc.Net.Client.Balancer.Subchannel[11]
      Subchannel id '1' state changed to Idle. Detail: 'Disconnected.'.
dbug: Grpc.Net.Client.Balancer.Subchannel[11]
      Subchannel id '1' state changed to Idle. Detail: 'Disconnected.'.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[3]
      Channel state updated to Idle.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[3]
      Channel state updated to Idle.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[3]
      Channel state updated to Idle.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[4]
      Channel picker updated.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[4]
      Channel picker updated.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[4]
      Channel picker updated.

Anything else we should know about your project / environment?

It seems the problem is related to use of GrpcClientFactory with multiple gRPC client types. It also seems like the problem is more likely the more gRPC client types that are registered.

That said, the problem is not always 100% reproduceable, so clearly there's some subtle timing specifics required to trigger the concurrency violation. I'd say something on the order of 1:10 runs results in the exception for me using the attached code on the hosts I have available to me. Sometimes it's a single exception raised per run, sometimes multiples.

The attached sample code is a very simplified version of the production code where this issues was first observed. It was hastily constructed for illustration purposes only.

GreeterServer.zip

[JamesNK]

Thanks for the detailed bug report. A fix will be in the next version of Grpc.Net.Client.

I updated to <PackageReference Include="Grpc.Net.Client" Version="2.46.0" />, but am still receiving the errors:

System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.set_Item(TKey key, TValue value)
   at Grpc.Net.Client.Configuration.ConfigObject.SetValue[T](String key, T value)
   at Grpc.Net.Client.Internal.Configuration.ConfigProperty`2.SetValue(ConfigObject inner, TValue value)
   at Grpc.Net.Client.Internal.Configuration.ConfigProperty`2.GetValue(ConfigObject inner)
   at Grpc.Net.Client.Configuration.ServiceConfig.get_LoadBalancingConfigs()
   at Grpc.Net.Client.Balancer.Internal.ConnectionManager.OnResolverResult(ResolverResult result)
   at Grpc.Net.Client.Balancer.StaticResolver.Start(Action`1 listener)
   at Grpc.Net.Client.Balancer.Internal.ConnectionManager.EnsureResolverStartedAsync()
   at Grpc.Net.Client.Balancer.Internal.ConnectionManager.ConnectAsync(Boolean waitForReady, CancellationToken cancellationToken)
   at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)

[JamesNK]

Strange. I'm not sure how this could still be happening. I've added a lock at a lower level that should prevent this error.

In using the sample app I provided above in an attempt reproduce the error with 2.46.0 just now, as means to try and further validate the status of the issue and gather more info, I've been unable to generate the error after several executions.

When I provided the initial feedback above, I was using a different (real-world) app as a test mechanism. It's entirely possible I was too hasty in reporting the continued trouble and overlooked some nuance that misled me into thinking the trouble was still there with 2.46.0.

Unfortunately, I won't be able to resume analyzing until early next week. Feel free to re-close this Issue. I'll only report additional feedback if I can determine a set of conditions where the problem remains. Otherwise, safe to assume I overlooked some nuance that misled me into believing the problem had not been fully resolved when in fact it was.

Thanks.

OK, I took a few minutes just now to do some basic sanity-checking and discovered that although the package reference in the cspoj refers to 2.46.0, the file actually on disk at the distribution point is still 2.45.0.

Not sure how that happened, using Visual Studio's Publish feature for distribution. The project exe and dll are up-to-date, but some of the nuget pkgs are stale.

You can definitely go ahead and re-close this Issue - other testing I've done shows no evidence of the original problem remaining in 2.46.0 (when the actual 2.46.0 dll is in fact in place). Sorry for the head fake and thanks for the fix.

[JamesNK]

Ok, thanks for checking. These things happen from time to time.

The additional change I made is still a good thing to do. Merging that PR will auto-close this issue.