Unable to cast object of type 'Microsoft.Data.ProviderBase.DbConnectionClosedConnecting' to type 'Microsoft.Data.SqlClient.SqlInternalConnectionTds'

Describe the bug

When I start the NET CORE API, and refresh my website client. I got this exception

System.InvalidCastException: Unable to cast object of type 'Microsoft.Data.ProviderBase.DbConnectionClosedConnecting' to type 'Microsoft.Data.SqlClient.SqlInternalConnectionTds'.
   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
   at Microsoft.Data.SqlClient.SqlConnection.Open()

To reproduce

Not sure how to reproduce. I see this exception happened when the API is just starting, or the API wakup after idle a long time

Expected behavior

A clear and concise description of what you expected to happen.

Further technical details

NET 8
SQL Server 2022
IIS 10

@mrvogiacu

@mrvogiacu Can you provide any steps to reproduce this issue? Can you mention the specific API when you are seeing this issue?

@apoorvdeshmukh

Hi @apoorvdeshmukh ,

It's really hard to reproduce it.
As I said, sometimes I get it when I start the local API and refresh my local website, it calls the local API, and I get this exception in the log file. In the production hosting, when the IIS sleeps because no requests, and I access the production website, I get the same exception. But if I continue calling the API, I never get this exception

More information, I set up the DI for the SQL class with the scope is Singleton.
For now, this bug is resolved when I remove the SQL DI and create the dynamic SQL instance each time I call to DB

@mrvogiacu

@mrvogiacu Thank you for the response. I'll discuss this internally with the team and get back to you.
Meanwhile, a search on this exception returned following threads. Does any of the workaround discussed in these threads work for you?
dotnet/efcore#11595
DevExpress/DevExtreme.AspNet.Data#352 (comment)

@apoorvdeshmukh

Hi @apoorvdeshmukh ,
I try to research and not see the link you gave me above. I take a look, and all of them talk about EF, but I am using Dapper with store procedure.

I guess the problem can be DI, because after I stopped using DI and let the SQL instance be created by my DA class, I did not get this exception anymore.
Before this exception, I was already using Scoped DI and got another problem :D

Analysis

After investigating the codebase, this InvalidCastException originates from a TOCTOU (time-of-check-time-of-use) race condition in SqlConnection.TryOpenInner().

Where it crashes

In SqlConnection.cs line 2228:

private bool TryOpenInner(TaskCompletionSource<DbConnectionInternal> retry)
{
    // ... TryOpenConnection succeeds, sets _innerConnection to SqlConnectionInternal ...

    var tdsInnerConnection = (SqlConnectionInternal)InnerConnection;  // ← CRASH: reads _innerConnection again
}

The race

Thread A calls Open() → TryOpenInner() → TryOpenConnection() completes successfully, setting _innerConnection to a valid SqlConnectionInternal.
Thread B concurrently calls Open() (or another state-changing operation) on the same SqlConnection instance, setting _innerConnection to DbConnectionClosedConnecting (a singleton transitional state marker).
Thread A reads InnerConnection for the cast — but now it's DbConnectionClosedConnecting, which is not assignable to SqlConnectionInternal → InvalidCastException.

The key issue is that _innerConnection is read twice without synchronization: once for TryOpenConnection() and again for the cast. The field is modified via both Interlocked.CompareExchange (in SetInnerConnectionFrom) and plain assignment (in SetInnerConnectionTo), so there is no consistent locking.

Why it happens after idle / on startup

IIS app pool recycling or connection pool pruning invalidates idle connections
New requests trigger concurrent Open() calls on shared connection instances
The pool state transitions can race with new connection attempts

Most likely user-side cause

SqlConnection is not thread-safe. This exception almost always means the same SqlConnection instance is being used from multiple threads simultaneously. Common causes in ASP.NET Core:

SqlConnection registered as Singleton in DI (should be Scoped or Transient)
A static SqlConnection field shared across requests
Missing await on async operations, causing overlapping operations on the same connection

Recommendation

Immediate fix: Ensure each database operation creates its own SqlConnection:

// ✅ Correct: new connection per operation
using var connection = new SqlConnection(connectionString);
await connection.OpenAsync();

// ❌ Wrong: shared/singleton connection
services.AddSingleton<SqlConnection>(...);

Library-side improvement

The TryOpenInner method could be hardened to detect this state and throw a more descriptive error (e.g., "Connection state changed unexpectedly. SqlConnection is not thread-safe — do not share instances across threads.") instead of the opaque InvalidCastException.

Could you confirm:

How is your SqlConnection created/managed? (DI registration? static field? per-request?)
Which version of Microsoft.Data.SqlClient are you using? (The stack trace references SqlInternalConnectionTds which has been renamed to SqlConnectionInternal in recent versions.)

Draft PR with reproducing tests: #4179

@paulmedynski

hi @paulmedynski,

When I got this exception:

I use DI with Singleton Scope
The Microsoft.Data.SqlClient version is 6.0.2

What I've done to fix this exception:

Instead of using Sql instance with the Singleton scope, my DA class (DI with Scoped scope) inherits the Sql class and creates a Sql instance for each request.

Add tests confirming InvalidCastException race in TryOpenInner (#3314)

Fix TryOpenInner InvalidCast race and add unit regression tests (#3314)…

Navigation Menu

Repository navigation

Unable to cast object of type 'Microsoft.Data.ProviderBase.DbConnectionClosedConnecting' to type 'Microsoft.Data.SqlClient.SqlInternalConnectionTds' #3314

Describe the bug

To reproduce

Expected behavior

Further technical details

10 remaining items

Analysis

Where it crashes

The race

Why it happens after idle / on startup

Most likely user-side cause

Recommendation

Library-side improvement

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Unable to cast object of type 'Microsoft.Data.ProviderBase.DbConnectionClosedConnecting' to type 'Microsoft.Data.SqlClient.SqlInternalConnectionTds' #3314

Description

Describe the bug

To reproduce

Expected behavior

Further technical details

Activity

apoorvdeshmukh commented on May 5, 2025

mrvogiacu commented on May 5, 2025

apoorvdeshmukh commented on May 5, 2025

mrvogiacu commented on May 5, 2025

10 remaining items

paulmedynski commented on Apr 10, 2026

Analysis

Where it crashes

The race

Why it happens after idle / on startup

Most likely user-side cause

Recommendation

Library-side improvement

mrvogiacu commented on Apr 12, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions