Concurrent first requests to MapGraphQLHttp and MapGraphQLSchema endpoints corrupt the executor's FeatureCollection #10480

Product

Hot Chocolate

Version

16.7.0-p.17 (also 16.6.7)

Link to minimal reproduction

Zip below

Steps to reproduce

Repro zip: HotChocolateProxyRace.zip

1. Run dotnet run -c Release in the repro (.NET 10 SDK).
2. The program repeatedly starts a fresh app on TestServer with MapGraphQLHttp("/graphql") and 16 MapGraphQLSchema endpoints, sends the first request to all of them at once, and stops at the first failed request. It usually fails within a few thousand iterations (seconds).
   With a single schema endpoint, as in a typical app (dotnet run -c Release --property:HotChocolateVersion=16.6.7 -- 1), it failed within a few hundred iterations.

What is expected?

All requests succeed.

What is actually happening?

Occasionally one of the first requests fails with HTTP 500, either because the executor's feature dictionary is corrupted (Operations that change non-concurrent collections must have exclusive access) or because the write was lost (Feature 'HotChocolate.AspNetCore.ExecutorSession' is not present).

Relevant log output

System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
   at System.Collections.Generic.Dictionary`2.set_Item(TKey key, TValue value)
   at HotChocolate.Features.FeatureCollection.set_Item(Type key, Object value)
   at HotChocolate.Features.FeatureCollection.Set[TFeature](TFeature instance)
   at HotChocolate.AspNetCore.HttpRequestExecutorProxy.OnConfigureRequestExecutor(IRequestExecutor newExecutor, IRequestExecutor oldExecutor)
   at HotChocolate.Execution.RequestExecutorProxy.GetExecutorAsync(CancellationToken cancellationToken)
   at HotChocolate.AspNetCore.HttpRequestExecutorProxy.GetOrCreateSessionAsync(CancellationToken cancellationToken)
   at HotChocolate.AspNetCore.HttpGetSchemaMiddleware.InvokeAsync(HttpContext context)
   at HotChocolate.AspNetCore.MiddlewareFactory.<>c__DisplayClass0_0.<<CreateCancellationMiddleware>b__1>d.MoveNext()

System.InvalidOperationException: Feature 'HotChocolate.AspNetCore.ExecutorSession' is not present.
   at HotChocolate.Features.FeatureCollectionExtensions.GetRequired[TFeature](IFeatureCollection featureCollection)
   at HotChocolate.AspNetCore.HttpRequestExecutorProxy.GetOrCreateSessionAsync(CancellationToken cancellationToken)
   at HotChocolate.AspNetCore.HttpGetSchemaMiddleware.InvokeAsync(HttpContext context)

Additional context

Each MapGraphQLHttp/MapGraphQLSchema (and MapGraphQL, MapGraphQLWebSocket, …) call creates its own HttpRequestExecutorProxy with its own semaphore. On first use, each proxy's OnConfigureRequestExecutor writes its ExecutorSession into the shared executor.Features, a FeatureCollection backed by a plain Dictionary, so writes from different proxies are not synchronized. The executor swap path (OnUpdateRequestExecutorAsync, run per proxy) has the same pattern. #8674 fixed a related race within a single proxy.

We hit this in production (one MapGraphQLHttp and one MapGraphQLSchema) right after a deployment, when smoke tests sent their first requests to /graphql and /graphql/sdl at the same time.