Checksum's usage of a native hash algorithm resulting in thread starvation

[davkean]

Version Used:
Investigating thread starvation in Speedometer's CopyRichNotAccurate, I see Checksum's usage of CAPI SHA256 hash algorithm is one of the cause of the starvations. It appears the following code path instantiates large number of finalizable objects (ie the underlying SafeHandle) on multiple threads at the same in a short burst. This hits a lock in the CLR's CFinalize::RegisterForFinalization causing contention and resulting in a lot of blocked time on the thread pool, 10% of the blocked time on these threads is this bug.

Steps to Reproduce:

1. Download trace from any run of Speedometer's CopyRichNotAccurate tests (ask me offline for how to do this)
2. Look at blocked time via Thread Time Stacks under CFinalize::RegisterForFinalization.

I had a look at SHA265Cng and it has a similar path so is likely not the path forward. Maybe the managed version of the algorithm? Or alternative, a custom wrapper around CNG that avoids creating a finalizable safe handle over and over again?

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[CyrusNajmabadi]

@stephentoub any suggestions on a good way to utilize this api while avoiding this problem?

[sharwell]

Based on the time shown above and the fact that this is already correctly performing on a background thread, I'm OK with leaving this unchanged if any of the following is true:

• The underlying issue has already been fixed on .NET, and only affects .NET Framework scenarios
• An API change is needed to fix this, but the API isn't available on .NET Framework
• There is no available API to avoid this lock

[stephentoub]

On .NET 5+ you can use the one-shot methods... zero allocation (finalizable or otherwise):

These are the way to go when targeting .NET 5+ surface area.

If you're stuck on .NET Framework, you can at least use SHA256.Create(), which unless FIPS is enabled will create a SHA256Managed instance and, while there will be a lot of allocation:

I don't believe any of it will be finalizable (though I'd need to check).

It looks like IncrementalHash is being used by Roslyn. IncrementalHash is just a thin wrapper around the underlying HashAlgorithm instance, and it's choosing the CNG implementations provided by the OS:

so you could just do what it does manually with your SHA256 instance: each time you use it for a new operation, call Initialize; call TransformBlock for each block of data (with a null output), and then when you want the result call TransformFinalBlock with an empty array and access the resulting Hash.

[sharwell]

@CyrusNajmabadi Sounds like RoslynIncrementalHash is about to exist. :)

[CyrusNajmabadi]

Yup. I'll work on this today.