AddRemoveFromDifferentThreads<string>.ConcurrentStack benchmark hangs on ARM64

[adamsitnik]

The AddRemoveFromDifferentThreads<string>.ConcurrentStack benchmark hangs on Windows ARM64 (.NET 7 Preview 1).

Repro:

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py --architecture arm64 -f net7.0 --dotnet-versions 7.0.100-preview.1.22077.12  --filter *AddRemoveFromDifferentThreads*string*ConcurrentStack*

I was not able to attach a VS debugger to the live process as VS seems to not support it on ARM64 yet.

I've captured a dump and uploaded it here.

PerfView shows me that the process is stuck in the following loop:

https://github.com/dotnet/performance/blob/3edf4c3149f5c903777f13af9b08b562b1672302/src/benchmarks/micro/libraries/System.Collections/Concurrent/AddRemoveFromDifferentThreads.cs#L94-L100

Since AddRemoveFromDifferentThreads<int>.ConcurrentStack and AddRemoveFromDifferentThreads<string>.ConcurrentBag work fine I guess it's a codegen issue.

Hardware: Surface Pro X, Win 10, arm64

Please let me know if there is any way I could help.

cc @kunalspathak @AndyAyersMS @kouvel

Tagging subscribers to this area: @JulieLeeMSFT
See info in area-owners.md if you want to be subscribed.

Issue Details

---

The AddRemoveFromDifferentThreads<string>.ConcurrentStack benchmark hangs on Windows ARM64 (.NET 7 Preview 1).

Repro:

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py --architecture arm64 -f net7.0 --dotnet-versions 7.0.100-preview.1.22077.12  --filter *AddRemoveFromDifferentThreads*string*ConcurrentStack*

I was not able to attach a VS debugger to the live process as VS seems to not support it on ARM64 yet.

I've captured a dump and uploaded it here.

PerfView shows me that the process is stuck in the following loop:

https://github.com/dotnet/performance/blob/3edf4c3149f5c903777f13af9b08b562b1672302/src/benchmarks/micro/libraries/System.Collections/Concurrent/AddRemoveFromDifferentThreads.cs#L94-L100

Since AddRemoveFromDifferentThreads<int>.ConcurrentStack and AddRemoveFromDifferentThreads<string>.ConcurrentBag work fine I guess it's a codegen issue.

Hardware: Surface Pro X, Win 10, arm64

Please let me know if there is any way I could help.

cc @kunalspathak @AndyAyersMS @kouvel

Author:	adamsitnik
Assignees:	-
Labels:	arch-arm64, os-windows, area-CodeGen-coreclr
Milestone:	-

[kunalspathak]

I will take a look at the code gen.

[adamsitnik]

@janvorli has hit this issue on Linux, I am going to rename the issue and change the labels

[janvorli]

What I can see on my Arm64 linux is that at the time of the hang, the stack is empty (the _head member of the stack is null), but we have read only 73033 elements (based on the count) while the Size is 200000.
It sounds like a concurrency issue between the ConcurrentStack.Push and ConcurrentStack.TryPop, might be due to some arm64 specific JIT optimization or something. Lock free code is always prone to various subtle details.

[EgorBo]

Does it reproduce every run? We can try to revert all the recent VOLATILE-related changes which I believe were correct but slightly changed things.

[janvorli]

I don't know, I just hit that while running the whole benchmark suite and I have attached lldb to the process to see what's wrong.
@adamsitnik might know details on the reproducibility.

[janvorli]

I've attempted to run the test (with the filter set as @adamsitnik described in the repro) five times and it hung in all of the cases. So it sounds like it reproduces with at least high probability.

[EgorBo]

Thanks! I'll check a couple of Memory-model related commits