System.Data.SqlClient 4.8.2 throws TransactionAbortedException (the connection has been broken)

[MichaelLogutov]

Description

After updating System.Data.SqlClient version from old 4.6.1 to latest 4.8.2 we started to see period errors like this:

System.Transactions.TransactionAbortedException: The transaction has aborted. ---> System.InvalidOperationException: The requested operation cannot be completed because the connection has been broken.

   --- End of inner exception stack trace ---
   at System.Transactions.TransactionStateAborted.EndCommit(InternalTransaction tx)
   at System.Transactions.CommittableTransaction.Commit()
   at System.Transactions.TransactionScope.InternalDispose()
   at System.Transactions.TransactionScope.Dispose()
   ...

Those exceptions happening in pure ADO.NET code with TransactionScope created with ReadCommitted isolation level, TransactionScopeOption.Required, 1 minute timeout and TransactionScopeAsyncFlowOption.Enabled.

After rolling back to System.Data.SqlClient 4.8.1 those exceptions dissapeared but we saw another new type of exceptions we've never seen before from cassandra driver:

Exception while trying borrow a connection from a pool

System.Net.Sockets.SocketException: Transport endpoint is not connected
   at Cassandra.Connections.HostConnectionPool.GetExistingConnections()
   at Cassandra.Connections.HostConnectionPool.BorrowExistingConnection()
   at async Cassandra.Connections.HostConnectionPool.GetConnectionFromHostAsync(?)

Downgrading back to 4.6.1 seems to fixed all issues.

Configuration

It happened on aspnet core 3.1 web applications and console netcore 3.1 applications with latest 3.1 images both running in container on Linux.

Regression?

It works on System.Data.SqlClient 4.6.1

Other information

This issue happening on heavy load application and only on production. I can't reproduce this issue locally.
I wish I could find source for System.Data.SqlClient library to see the diff for 4.8.2 version.

Tagging subscribers to this area: @cheenamalhotra, @David-Engel
See info in area-owners.md if you want to be subscribed.

[Wraith2]

The System version the library has been frozen and moved out of this repository in favour of Microsoft.Data.SqlClient https://github.com/dotnet/SqlClient which can be versioned separately. It contains fixes and improvements beyond those present in the framework provided versions. If possible give it a try and see if the behaviour you're seeing has already been resolved.

[MichaelLogutov]

Does it requires rewriting all namespaces Microsoft.*? Because we can't afford such rework in our libraries and all microservices.

[Wraith2]

It does. You should probably consider introducing abstractions in your codebase to avoid such things causing agility limitations.

[MichaelLogutov]

Well, it's not that simple when you have large code base and 20 product team developers.

Can I see the source code or at least changelog of what been changed in 4.8.2?

[devsko]

Maybe these help
https://github.com/microsoft/dotnet/tree/master/releases
https://referencesource.microsoft.com/#System.Data

[MichaelLogutov]

Thanks. I can see the source now but I can't seems to see the changes in 4.8.2
https://github.com/microsoft/dotnet/tree/master/releases do not seems to have release notes for individual nuget packages like System.Data.SqlClient. Or I'm missing something?

[Wraith2]

It's part of the framework so it doesn't get it's own update schedule. You probably want to review all the pull requests which targeted files in that project.

[MichaelLogutov]

I couldn't find the source code in github for System.Data.SqlClient. I saw only System.Data.Common

[devsko]

AFAIK System.Data.SqlClient is not open sourced but source open at https://referencesource.microsoft.com/#System.Data as mentioned above.

[Wraith2]

AFAIK System.Data.SqlClient is not open sourced but source open at

It absolutely was and was part of this repository. It was removed in favour of the Microsoft version as I said earlier. To see the code you'll have to look into the history prior to it's removal.

[devsko]

@Wraith2 Just curious. It was the same code for core / desktop? Where lives the System.Data.SqlClient desktop code now?

[Wraith2]

The desktop version was distinct from the core version. This repo only contained the core version.
The code is still here but not current which I why I say you'll need to look through the repo history to look at it. You can look back at any file that has ever existed by reviewing the git history but because it isn't currently being build and shipped it's not visible in the current master.

The code was forked into Microsoft.Data.SqlClient which also contains the desktop version. The Microsoft version supercedes the System version and is versioned independently so it can be updated without a framework update. It contains performance improvements new features and bug fixes that may not be ported back to core or desktop verisons.

[devsko]

Thanks.
I totally misunderstood this issue. After reading 4.6.1 and 4.8.2 I thought it is all about .NET Framework - but it isn't. Sorry for the confusion.

[cheenamalhotra]

Tagging @saurabh500 @corivera for System.Data.SqlClient (.Net Framework)