SortedDictionary<BigInteger, MyObject> System.AccessViolationException

[swtrse]

Description

HI,

I ran into a problem were using SortedDictionary<BigInteger, MyObject> ran into a System.AccessViolationException every single time.
The error occured during an BenchmarkDotNet run.
The error does not occur in debug mode. As far as I found out the error only appeared in Release mode.
Also, the error disappears if you want to analyze the behavior with dotMemory.

Reproduction Steps

MyObject is a sealed generic class bit I think it does not matter.

public sealed class Test
{
    private readonly SortedDictionary<BigInteger, MyObject> _limitSortedList;

    .......

    private MyObject? GetNextNode(BigInteger target)
    {
	if (_limitSortedList.TryGetValue(target, out Swtrse2Node<TKey, TOrder>? node)) { return node; }

	var low = 0;
	int high = _limitSortedList.Count - 1;
	List<BigInteger> list = _limitSortedList.Keys.ToList();   //<---- Exception

	while (low <= high)
	{
		int mid = (low + high) / 2;
		int comparison = _limitSortedList.Comparer.Compare(list[mid], target);

		if (comparison < 0) { low = mid + 1; }
		else { high = mid - 1; }
	}

	return low < _limitSortedList.Count ? _limitSortedList[list[low]] : null;
    }
}

Expected behavior

I expect no AccessViolationException when calling Framework functions

Actual behavior

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
at System.Collections.Generic.List1[[System.Numerics.BigInteger, System.Runtime.Numerics, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]]..ctor(System.Collections.Generic.IEnumerable1<System.Numerics.BigInteger>)

Regression?

No response

Known Workarounds

No response

Configuration

// BenchmarkDotNet v0.14.0
// Runtime=.NET 9.0.0 (9.0.24.43107), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
// GC=Concurrent Workstation
// HardwareIntrinsics=AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT VectorSize=256

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-collections
See info in area-owners.md if you want to be subscribed.

[EgorBo]

Please attach a self-contained repro or a crash dump, it's not possible to diagnose the issue from just the given snippet. It could be caused by a semi-valid unsafe/interop in your application (stack traces could be confusing in such cases) or a thread-safety issue (SortedDictionary is not thread safe). Also, is this Linux or Windows? BDN has a known issue on Linux + DisassemblyDiagnoser.

[swtrse]

Problem is. I tried using dotMemory to analyse the error. As soon as the memory profiler is attached the error is gone.
I see no option to generate a crashdump in any way.

I run on Windows 11.
I do not use any unsafe code.

[jkotas]

I see no option to generate a crashdump in any way.

You can enable crashdumps by following https://learn.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps (use FullDump - it has the complete info)

[swtrse]

Thx here is the dump https://we.tl/t-zWlR3MhCbz. Sorry I had to use wetransfer ,github is blocking the upload.

And I strongly believe the Error is within SortedDictionary<>.Keys in combination with ToList() or ToArray()
Justification:

1. If I change the code to a version without calling ToList() or ToArray() the error is gone
2. If I change SortedDictionary<> to SortedList<> the error is gone
3. The error happens instantly after starting the application. I suspect on one of the first calls.

[jkotas]

Details of the crash - bad GC pointer:

System_Collections!System.Collections.Generic.SortedSet<System.Collections.Generic.KeyValuePair<System.Numerics.BigInteger,System.__Canon>>.InOrderTreeWalk+0x135:
00007ffe`b79fdbb5 ff4514          inc     dword ptr [rbp+14h] ss:0000013a`1638abfc=00002213

The crash is on .NET 9 RC1. We fixed several bugs that can lead to intermittent crashes like this one for .NET 9 RC2. Could you please upgrade to .NET 9 RC2 and see whether it is going to fix the crash?

Thx here is the dump https://we.tl/t-zWlR3MhCbz.

Next time, you can create an issue at https://developercommunity.visualstudio.com/, attach the dump to it and link it from here. developercommunity does not have uploads limits like github and it has better privacy controls.

[swtrse]

The crash still remains with .Net 9 RC2.

[jkotas]

Could you please upload a .NET 9 RC2 crash dump?

[swtrse]

I uploaded the crashdump here:
https://developercommunity.visualstudio.com/t/SortedDictionary-SystemAccessViolationE/10769841

[jkotas]

Thanks - the RC2 crash is exactly same as the RC1 crash.

There is weird discrepancy between the state of the registers and the faulting address in EXCEPTION_RECORD in both dumps.

00007ffe`b79fdbb5 ff4514          inc     dword ptr [rbp+14h] ss:0000013a`1638abfc=00002213

vs.

0:024> dx -r1 ((coreclr!_EXCEPTION_RECORD *)0x38c4efc4f0)
((coreclr!_EXCEPTION_RECORD *)0x38c4efc4f0)                 : 0x38c4efc4f0 [Type: _EXCEPTION_RECORD *]
    [+0x000] ExceptionCode    : 0xc0000005 [Type: unsigned long]
    [+0x004] ExceptionFlags   : 0x1 [Type: unsigned long]
    [+0x008] ExceptionRecord  : 0x0 [Type: _EXCEPTION_RECORD *]
    [+0x010] ExceptionAddress : 0x7ffc351acd62 [Type: void *]
    [+0x018] NumberParameters : 0x2 [Type: unsigned long]
    [+0x020] ExceptionInformation [Type: unsigned __int64 [15]]
0:024> dx -r1 (*((coreclr!unsigned __int64 (*)[15])0x38c4efc510))
(*((coreclr!unsigned __int64 (*)[15])0x38c4efc510))                 [Type: unsigned __int64 [15]]
    [0]              : 0x1 [Type: unsigned __int64]
    [1]              : 0x24eba52b1a6 [Type: unsigned __int64] <- This should be same as [rbp+14h] ss:0000013a`1638abfc

This suggests that hardware somehow accessed a different address than what the registers are pointing at.

Are you able to reproduce this crash on more than one machine? One possible explanation of these symptoms is a hardware defect.

[jkotas]

Also, could you please try whether it reproduces with disabled tiered compilation (e.g. DOTNET_TieredCompilation=0 environment variable)?

The faulting method has 4 OptimizedTier1OSR variants of the code that is unusual. It may be related.

  IL Addr:            00007fff8c6cc624
     CodeAddr:           00007ffeb79e5ca0  (OptimizedTier1OSR)
     NativeCodeVersion:  0000013A11EB5F00
     CodeAddr:           00007ffeb79e5fa0  (OptimizedTier1OSR)
     NativeCodeVersion:  0000013A11EB5C40
     CodeAddr:           00007ffeb79fcb40  (QuickJitted + Instrumented)
     NativeCodeVersion:  0000013A0078C9D0
     CodeAddr:           00007ffeb79fd220  (OptimizedTier1OSR)
     NativeCodeVersion:  0000013A11EB6180
     CodeAddr:           00007ffeb79fda80  (OptimizedTier1OSR)
     NativeCodeVersion:  0000013A11EB5AC0
     CodeAddr:           0000000000000000  (OptimizedTier1)
     NativeCodeVersion:  0000013A00794FC0
     CodeAddr:           00007ffeb79e3e50  (QuickJitted)
     NativeCodeVersion:  0000000000000000

[swtrse]

Also, could you please try whether it reproduces with disabled tiered compilation (e.g. DOTNET_TieredCompilation=0 environment variable)?

I tried that with no effect, The error is still the same

[swtrse]

This suggests that hardware somehow accessed a different address than what the registers are pointing at.

Are you able to reproduce this crash on more than one machine? One possible explanation of these symptoms is a hardware defect.

Thats highly unlikely. The system despite using windows 11 as an OS runs rock solid for weeks without reboot. A Hardware defect in this kind of area should have a negative impact on system stability in general, but I do not see it.

The error did not appear on a second machine I tried. But the other machine is a complexly different CPU generation. Maybe it is something CPU or chipset specific.

I did backport the solution to .net 8 as soon as the project runs with .net8 the error is gone. Even when Benchmark.net running with .net 9 environment

// Benchmark Process Environment Information:
// BenchmarkDotNet v0.14.0
// Runtime=.NET 8.0.10 (8.0.1024.46610), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
// GC=Concurrent Workstation
// HardwareIntrinsics=AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT VectorSize=256
// Job: Job-NHHCHO(Runtime=.NET 9.0, Toolchain=InProcessEmitToolchain, InvocationCount=1, IterationCount=3, LaunchCount=1, UnrollFactor=1, WarmupCount=3)

I can provide the whole solution if that is of any help.