Concurrent connection usage and MARS

[roji]

As part of running some compatibility checks between EF Core 6.0 and SqlClient 3.0.0 and 4.0.0-preview1.21237.2 (dotnet/efcore#25766), I came across what looks like a considerable perf regression. While the EF test suite completed in around 4m30s with SqlClient 3.0.0, it took around 12 minutes with 4.0.0-preview1.21237.2.

Some digging seems to point at test QueryBugsTest.Thread_safety_in_relational_command_cache; with 3.0.0 the total run completes in 20 seconds (including all startup), whereas with 4.0.0-preview1.21237.2 it takes around 5m15s. This test runs 100 parallel queries - no big data is being transferred, and purely sync I/O is used.

To run this, you can check out the EF Core code base, run build.cmd to get the latest dotnet core, go inside test/EFCore.SqlServer.FunctionalTests and run dotnet test --filter Thread_safety_in_relational_command_cache

[Wraith2]

Windows or Linux will be an important factor.

[roji]

Good point, this happened on my machine running on Ubuntu 21.04.

[ajcvickers]

I will do a test run on my Windows laptop.

[ajcvickers]

I can't get any of the SQL Server tests to run on my machine using 4.0.0-preview1.21237.2. I get:

System.ComponentModel.Win32Exception : The certificate chain was issued by an authority that is not trusted.

Is there something else I need to configure on my machine to make these tests run?

Full output:

Test run for C:\dotnet\efcore\artifacts\bin\EFCore.SqlServer.FunctionalTests\Debug\net6.0\Microsoft.EntityFrameworkCore.SqlServer.FunctionalTests.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.0.0-preview-20210817-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:09.14]     Microsoft.EntityFrameworkCore.Query.QueryBugsTest.Thread_safety_in_relational_command_cache [FAIL]
  Failed Microsoft.EntityFrameworkCore.Query.QueryBugsTest.Thread_safety_in_relational_command_cache [346 ms]
  Error Message:
   Microsoft.Data.SqlClient.SqlException : A connection was successfully established with the server, but then an error occurred during the login process. (provider: SSL Provider, error: 0 - The certificate chain was issued by an authority that is not trusted.)
---- System.ComponentModel.Win32Exception : The certificate chain was issued by an authority that is not trusted.
  Stack Trace:
     at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParserStateObject.ThrowExceptionAndWarning(Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParserStateObject.SNIWritePacket(PacketHandle packet, UInt32& sniError, Boolean canAccumulate, Boolean callerHasConnectionLock)
   at Microsoft.Data.SqlClient.TdsParserStateObject.WriteSni(Boolean canAccumulate)
   at Microsoft.Data.SqlClient.TdsParserStateObject.WritePacket(Byte flushMode, Boolean canAccumulate)
   at Microsoft.Data.SqlClient.TdsParser.TdsLogin(SqlLogin rec, FeatureExtension requestedFeatures, SessionData recoverySessionData, FederatedAuthenticationFeatureExtensionData fedAuthFeatureExtensionData)
   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.Login(ServerInfo server, TimeoutTimer timeout, String newPassword, SecureString newSecurePassword)
   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover)
   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)
   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)
   at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool)
   at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
   at Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
   at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
   at Microsoft.Data.SqlClient.SqlConnection.Open()
   at Microsoft.EntityFrameworkCore.TestUtilities.SqlServerTestStore.ExecuteCommand[T](DbConnection connection, Func`2 execute, String sql, Boolean useTransaction, Object[] parameters) in C:\dotnet\efcore\test\EFCore.SqlServer.FunctionalTests\TestUtilities\SqlServerTestStore.cs:line 330
   at Microsoft.EntityFrameworkCore.TestUtilities.SqlServerTestStore.<>c__37`1.<Execute>b__37_0(<>f__AnonymousType83`5 state) in C:\dotnet\efcore\test\EFCore.SqlServer.FunctionalTests\TestUtilities\SqlServerTestStore.cs:line 316
   at Microsoft.EntityFrameworkCore.ExecutionStrategyExtensions.<>c__DisplayClass12_0`2.<Execute>b__0(DbContext c, TState s) in C:\dotnet\efcore\src\EFCore\Storage\ExecutionStrategyExtensions.cs:line 341
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.<>c__DisplayClass28_0`2.<Execute>b__0(DbContext context, TState state) in C:\dotnet\efcore\src\EFCore\Storage\ExecutionStrategy.cs:line 161
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementation[TState,TResult](Func`3 operation, Func`3 verifySucceeded, TState state) in C:\dotnet\efcore\src\EFCore\Storage\ExecutionStrategy.cs:line 176
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded) in C:\dotnet\efcore\src\EFCore\Storage\ExecutionStrategy.cs:line 160
   at Microsoft.EntityFrameworkCore.ExecutionStrategyExtensions.Execute[TState,TResult](IExecutionStrategy strategy, TState state, Func`2 operation, Func`2 verifySucceeded) in C:\dotnet\efcore\src\EFCore\Storage\ExecutionStrategyExtensions.cs:line 339
   at Microsoft.EntityFrameworkCore.ExecutionStrategyExtensions.Execute[TState,TResult](IExecutionStrategy strategy, TState state, Func`2 operation) in C:\dotnet\efcore\src\EFCore\Storage\ExecutionStrategyExtensions.cs:line 288
   at Microsoft.EntityFrameworkCore.TestUtilities.SqlServerTestStore.Execute[T](DbConnection connection, Func`2 execute, String sql, Boolean useTransaction, Object[] parameters) in C:\dotnet\efcore\tes\TestUtilities\SqlServerTestStore.cs:line 247
   at Microsoft.EntityFrameworkCore.TestUtilities.SqlServerTestStore.CreateDatabase(Action`1 clean) in C:\dotnet\efcore\test\EFCore.SqlServer.FunctionalTests\TestUtilities\SqlServerTestStore.cs:line 110
   at Microsoft.EntityFrameworkCore.TestUtilities.SqlServerTestStore.Initialize(Func`1 createContext, Action`1 seed, Action`1 clean) in C:\dotnet\efcore\test\EFCore.SqlServer.FunctionalTests\TestUtilities\SqlServerTestStore.cs:line 86
.cs:line 52
   at Microsoft.EntityFrameworkCore.TestUtilities.TestStore.Initialize(IServiceProvider serviceProvider, Func`1 createContext, Action`1 seed, Action`1 clean) in C:\dotnet\efcore\test\EFCore.Specification.Tests\TestUtilities\TestStore.cs:line 45
   at Microsoft.EntityFrameworkCore.TestUtilities.RelationalTestStore.Initialize(IServiceProvider serviceProvider, Func`1 createContext, Action`1 seed, Action`1 clean) in C:\dotnet\efcore\test\EFCore.Relational.Specification.Tests\TestUtilities\RelationalTestStore.cs:line 43
   at Microsoft.EntityFrameworkCore.TestUtilities.SqlServerTestStore.InitializeSqlServer(IServiceProvider serviceProvider, Func`1 createContext, Action`1 seed) in C:\dotnet\efcore\test\EFCore.SqlServer.FunctionalTests\TestUtilities\SqlServerTestStore.cs:line 76
   at Microsoft.EntityFrameworkCore.TestUtilities.SqlServerTestStore.CreateInitialized(String name, Boolean useFileName, Nullable`1 multipleActiveResultSets) in C:\dotnet\efcore\test\EFCore.SqlServer.FunctionalTests\TestUtilities\SqlServerTestStore.cs:line 44
   at Microsoft.EntityFrameworkCore.Query.QueryBugsTest.CreateTestStore() in C:\dotnet\efcore\test\EFCore.SqlServer.FunctionalTests\Query\QueryBugsTest.cs:line 10401
   at Microsoft.EntityFrameworkCore.NonSharedModelTestBase.Initialize[TContext](Action`1 onModelCreating, Action`1 onConfiguring, Action`1 addServices, Func`2 shouldLogCategory, Func`1 createTestStore, Boolean usePooling) in C:\dotnet\efcore\test\EFCore.Specification.Tests\NonSharedModelTestBase.cs:line 86
   at Microsoft.EntityFrameworkCore.NonSharedModelTestBase.InitializeAsync[TContext](Action`1 onModelCreating, Action`1 onConfiguring, Action`1 addServices, Action`1 seed, Func`2 shouldLogCategory, Func`1 createTestStore, Boolean usePooling) in C:\dotnet\efcore\test\EFCore.Specification.Tests\NonSharedModelTestBase.cs:line 67
   at Microsoft.EntityFrameworkCore.Query.QueryBugsTest.Thread_safety_in_relational_command_cache() in C:\dotnet\efcore\test\EFCore.SqlServer.FunctionalTests\Query\QueryBugsTest.cs:line 7092
--- End of stack trace from previous location ---
----- Inner Stack Trace -----


Failed!  - Failed:     1, Passed:     0, Skipped:     0, Total:     1, Duration: < 1 ms - Microsoft.EntityFrameworkCore.SqlServer.FunctionalTests.dll (net6.0)

[roji]

Yes, you need to add Encrypt=false to your connection string; SqlClient 4.0.0 has Encrypt set to true by default (#1210).

[Wraith2]

Preview 4 has a security change which forces encryption and trust. You either need to disable encryption (not recommended because it'll be what users have enabled) or add TrustServerCertificate=True; to your connection string.

[ajcvickers]

Not seeing a regression on Windows:

With 2.1:

Test run for C:\dotnet\efcore\artifacts\bin\EFCore.SqlServer.FunctionalTests\Debug\net6.0\Microsoft.EntityFrameworkCore.SqlServer.FunctionalTests.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.0.0-preview-20210817-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: < 1 ms - Microsoft.EntityFrameworkCore.SqlServer.FunctionalTests.dll (net6.0)

With 4.0 preview:

Test run for C:\dotnet\efcore\artifacts\bin\EFCore.SqlServer.FunctionalTests\Debug\net6.0\Microsoft.EntityFrameworkCore.SqlServer.FunctionalTests.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.0.0-preview-20210817-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: < 1 ms - Microsoft.EntityFrameworkCore.SqlServer.FunctionalTests.dll (net6.0)

[Wraith2]

Ok, Managed SNI it is. @roji were your tests with Encryption off? and do you know if MARS was on or off?

[roji]

Yeah, Encrypt=false and no MARS. Can also run any other tests here - just let me know.

[Wraith2]

No Mars is bad news. That means it's not just the usual threadpool exhaustion because with mars off sync IO is really synchronous in managed SNI.

[ajcvickers]

Okay, so ran with managed SNI on Windows using:

static SqlServerTestStore()
{
    var ManagedNetworkingAppContextSwitch = "Switch.Microsoft.Data.SqlClient.UseManagedNetworkingOnWindows";
    AppContext.SetSwitch(ManagedNetworkingAppContextSwitch, true);
}

My connection string is:

Server=localhost;Database=master;Trusted_Connection=True;TrustServerCertificate=True;

Now the test runs for about 10-15 seconds, but then consistently fails with:

Microsoft.EntityFrameworkCore.Query.QueryBugsTest.Thread_safety_in_relational_command_cache

Microsoft.Data.SqlClient.SqlException: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server...

Microsoft.Data.SqlClient.SqlException
A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
   at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParser.Connect(ServerInfo serverInfo, SqlInternalConnectionTds connHandler, Boolean ignoreSniOpenTimeout, Int64 timerExpire, Boolean encrypt, Boolean trustServerCert, Boolean integratedSecurity, Boolean withFailover, SqlAuthenticationMethod authType)
   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover)
   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)
   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)
   at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool)
   at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
   at Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
   at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
   at Microsoft.Data.SqlClient.SqlConnection.Open()
   at Microsoft.EntityFrameworkCore.TestUtilities.SqlServerTestStore.ExecuteCommand[T](DbConnection connection, Func`2 execute, String sql, Boolean useTransaction, Object[] parameters) in C:\dotnet\efcore\test\EFCore.SqlServer.FunctionalTests\TestUtilities\SqlServerTestStore.cs:line 336
   at Microsoft.EntityFrameworkCore.TestUtilities.SqlServerTestStore.<>c__38`1.<Execute>b__38_0(<>f__AnonymousType83`5 state) in C:\dotnet\efcore\test\EFCore.SqlServer.FunctionalTests\TestUtilities\SqlServerTestStore.cs:line 322
   at Microsoft.EntityFrameworkCore.ExecutionStrategyExtensions.<>c__DisplayClass12_0`2.<Execute>b__0(DbContext c, TState s) in C:\dotnet\efcore\src\EFCore\Storage\ExecutionStrategyExtensions.cs:line 341
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.<>c__DisplayClass28_0`2.<Execute>b__0(DbContext context, TState state) in C:\dotnet\efcore\src\EFCore\Storage\ExecutionStrategy.cs:line 161
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementation[TState,TResult](Func`3 operation, Func`3 verifySucceeded, TState state) in C:\dotnet\efcore\src\EFCore\Storage\ExecutionStrategy.cs:line 176
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded) in C:\dotnet\efcore\src\EFCore\Storage\ExecutionStrategy.cs:line 160
   at Microsoft.EntityFrameworkCore.ExecutionStrategyExtensions.Execute[TState,TResult](IExecutionStrategy strategy, TState state, Func`2 operation, Func`2 verifySucceeded) in C:\dotnet\efcore\src\EFCore\Storage\ExecutionStrategyExtensions.cs:line 339
   at Microsoft.EntityFrameworkCore.ExecutionStrategyExtensions.Execute[TState,TResult](IExecutionStrategy strategy, TState state, Func`2 operation) in C:\dotnet\efcore\src\EFCore\Storage\ExecutionStrategyExtensions.cs:line 288
   at Microsoft.EntityFrameworkCore.TestUtilities.SqlServerTestStore.Execute[T](DbConnection connection, Func`2 execute, String sql, Boolean useTransaction, Object[] parameters) in C:\dotnet\efcore\test\EFCore.SqlServer.FunctionalTests\TestUtilities\SqlServerTestStore.cs:line 313
   at Microsoft.EntityFrameworkCore.TestUtilities.SqlServerTestStore.ExecuteScalar[T](DbConnection connection, String sql, Object[] parameters) in C:\dotnet\efcore\test\EFCore.SqlServer.FunctionalTests\TestUtilities\SqlServerTestStore.cs:line 253
   at Microsoft.EntityFrameworkCore.TestUtilities.SqlServerTestStore.CreateDatabase(Action`1 clean) in C:\dotnet\efcore\test\EFCore.SqlServer.FunctionalTests\TestUtilities\SqlServerTestStore.cs:line 116
   at Microsoft.EntityFrameworkCore.TestUtilities.SqlServerTestStore.Initialize(Func`1 createContext, Action`1 seed, Action`1 clean) in C:\dotnet\efcore\test\EFCore.SqlServer.FunctionalTests\TestUtilities\SqlServerTestStore.cs:line 92
   at Microsoft.EntityFrameworkCore.TestUtilities.TestStore.<>c__DisplayClass13_0.<Initialize>b__1() in C:\dotnet\efcore\test\EFCore.Specification.Tests\TestUtilities\TestStore.cs:line 45
   at Microsoft.EntityFrameworkCore.TestUtilities.TestStoreIndex.CreateNonShared(String name, Action initializeDatabase) in C:\dotnet\efcore\test\EFCore.Specification.Tests\TestUtilities\TestStoreIndex.cs:line 52
   at Microsoft.EntityFrameworkCore.TestUtilities.TestStore.Initialize(IServiceProvider serviceProvider, Func`1 createContext, Action`1 seed, Action`1 clean) in C:\dotnet\efcore\test\EFCore.Specification.Tests\TestUtilities\TestStore.cs:line 45
   at Microsoft.EntityFrameworkCore.TestUtilities.RelationalTestStore.Initialize(IServiceProvider serviceProvider, Func`1 createContext, Action`1 seed, Action`1 clean) in C:\dotnet\efcore\test\EFCore.Relational.Specification.Tests\TestUtilities\RelationalTestStore.cs:line 43
   at Microsoft.EntityFrameworkCore.TestUtilities.SqlServerTestStore.InitializeSqlServer(IServiceProvider serviceProvider, Func`1 createContext, Action`1 seed) in C:\dotnet\efcore\test\EFCore.SqlServer.FunctionalTests\TestUtilities\SqlServerTestStore.cs:line 82
   at Microsoft.EntityFrameworkCore.TestUtilities.SqlServerTestStore.CreateInitialized(String name, Boolean useFileName, Nullable`1 multipleActiveResultSets) in C:\dotnet\efcore\test\EFCore.SqlServer.FunctionalTests\TestUtilities\SqlServerTestStore.cs:line 50
   at Microsoft.EntityFrameworkCore.Query.QueryBugsTest.CreateTestStore() in C:\dotnet\efcore\test\EFCore.SqlServer.FunctionalTests\Query\QueryBugsTest.cs:line 10401
   at Microsoft.EntityFrameworkCore.NonSharedModelTestBase.Initialize[TContext](Action`1 onModelCreating, Action`1 onConfiguring, Action`1 addServices, Func`2 shouldLogCategory, Func`1 createTestStore, Boolean usePooling) in C:\dotnet\efcore\test\EFCore.Specification.Tests\NonSharedModelTestBase.cs:line 86
   at Microsoft.EntityFrameworkCore.NonSharedModelTestBase.InitializeAsync[TContext](Action`1 onModelCreating, Action`1 onConfiguring, Action`1 addServices, Action`1 seed, Func`2 shouldLogCategory, Func`1 createTestStore, Boolean usePooling) in C:\dotnet\efcore\test\EFCore.Specification.Tests\NonSharedModelTestBase.cs:line 67
   at Microsoft.EntityFrameworkCore.Query.QueryBugsTest.Thread_safety_in_relational_command_cache() in C:\dotnet\efcore\test\EFCore.SqlServer.FunctionalTests\Query\QueryBugsTest.cs:line 7092
   at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_1.<<InvokeTestMethodAsync>b__1>d.MoveNext() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 264
--- End of stack trace from previous location ---
   at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\ExecutionTimer.cs:line 48
   at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 90

[roji]

Sounds very similar/related...

[Wraith2]

That does smell like threadpool exhaustion..
It'll start multiple connections and rely on a task callback to complete the connection but because the threadpool hasn't ramped up you haven't got enough threads to schedule the completion and when one becomes available as it is added to the threadpool it'll likely pick up a thread that's spend so long connecting that it's passed the timeout and will fail. If you make it through that phase things will sort of work. That's the exact presentation in #422 but it looks like it's been extended into the pool now.

The immediate easy test it to artificially inflate the threadpool minthreads to the number of connections you expect to open, in this case 100, and then see if it still fails. If it all works then we know what the problem is. Fixing it though, that's a different matter.

[ajcvickers]

Some more info: this fails in exactly the same way with SqlClient 2.1 on Windows with managed SNI. Increasing thread pool size doesn't make any difference.

static SqlServerTestStore()
{
    var ManagedNetworkingAppContextSwitch = "Switch.Microsoft.Data.SqlClient.UseManagedNetworkingOnWindows";
    AppContext.SetSwitch(ManagedNetworkingAppContextSwitch, true);
    ThreadPool.GetMaxThreads(out var workerThreads, out var completionPortThreads);
    ThreadPool.SetMaxThreads(workerThreads * 20, completionPortThreads * 20);
}