system.collections.concurrent.tests failed in CI

[runfoapp]

Failures 7/5-8/26 - Runfo for last 30 days:

Day	Run	Details
8/7-8/26		Around 150 failures were manually reviewed, none was the same
8/7	PR #70941 - PR 1928330	A QueueUserWorkItemCallback was never called! - Alpine.314.Arm32.Open
8/4	PR #73186 - 1924484	A QueueUserWorkItemCallback was never called! - Alpine.314.Arm32.Open
8/1	PR #72896 - 1916944	A QueueUserWorkItemCallback was never called! - Alpine.314.Arm32.Open
8/1	PR #73185 - 1916354	Most likely caused by the PR changes pStaticFieldDescs != NULL - 6x failures - all the same failures
8/1	PR #49576 - 1915051	Caused by the PR changes !IsTypeDesc() - 7x failures - all the same failures
7/31	PR #49576 - 1914778	Caused by the PR changes !IsTypeDesc() - 5x failures Assertion failed 'rv1 && !varTypeIsGC(rv1->TypeGet())' in 'System.Collections.Generic.ComparerHelpers:CreateDefaultEqualityComparer(System.Type):System.Object' during 'Optimize Valnum CSEs' (IL size 261; hash 0x6fa03044; Tier1) - 3x failures
7/30	PR #49576 - 1913943	Caused by the PR changes lpAddress || (dwSize 1.10864e-311_SystemInfo.dwAllocationGranularity) == 0 - 2x failures
7/30	PR #73095 - 1913236	Most likely caused by the PR changes A fatal error occurred. The folder [/root/helix/work/correlation/host/fxr] does not exist - 17x failures
7/29	PR #49576 - 1912700	Caused by the PR changes Consistency check failed: AV in clr at this callstack
7/29	PR #49576 - 1912185	Caused by the PR changes AV - 2x
7/28	PR #49576 - 1910538	Caused by the PR changes Consistency check failed: AV in clr at this callstack
7/28	Rolling run 1909685	A QueueUserWorkItemCallback was never called!
7/19	PR #72179 - 1889791	8x failures - unrelated to other failures
7/5-7/13	*	Logs are not available anymore

A QueueUserWorkItemCallback was never called!

Example: Console log

/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Collections.Concurrent.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Collections.Concurrent.Tests (found 885 of 893 test cases)
  Starting:    System.Collections.Concurrent.Tests (parallel test collections = on, max threads = 4)
Process terminated. Assertion failed.
A QueueUserWorkItemCallback was never called!
   at System.Threading.QueueUserWorkItemCallbackBase.Finalize() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 1175
./RunTests.sh: line 168:    24 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Collections.Concurrent.Tests.runtimeconfig.json --depsfile System.Collections.Concurrent.Tests.deps.json xunit.console.dll System.Collections.Concurrent.Tests.dll 

There are several failures of this test recently, but the errors vary and it is unclear if they are related to same root cause or not.

Runfo Tracking Issue: system.collections.concurrent.tests failed in CI

Build	Definition	Kind	Run Name	Console	Core Dump	Test Results	Run Client
1969744	runtime	PR 74623	net7.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1969744	runtime	PR 74623	net7.0-windows-Release-x64-CoreCLR_checked-Windows.10.Amd64.Open	console.log	core dump		runclient.py
1968656	runtime	PR 74623	net7.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1968656	runtime	PR 74623	net7.0-windows-Release-x64-CoreCLR_checked-Windows.10.Amd64.Open	console.log	core dump		runclient.py
1966436	runtime	PR 73968	net7.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1966436	runtime	PR 73968	net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1965131	runtime	PR 73968	net7.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1965131	runtime	PR 73968	net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1964207	runtime	PR 69578	net7.0-Linux-Debug-x64-mono_interpreter_release-(Debian.10.Amd64.Open)Ubuntu.1804.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-10-helix-amd64-20210304164434-56c6673	console.log	core dump		runclient.py
1964186	runtime	Rolling	net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3	console.log
1964186	runtime	Rolling	net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3	console.log
1964132	runtime	PR 74428	net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3	console.log
1964132	runtime	PR 74428	net7.0-Linux-Debug-arm64-Mono_release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20220427172132-97d8652	console.log
1964132	runtime	PR 74428	net7.0-Linux-Debug-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652	console.log
1964122	runtime	PR 74502	net7.0-Linux-Debug-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652	console.log
1964122	runtime	PR 74502	net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3	console.log
1964122	runtime	PR 74502	net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3	console.log
1961194	runtime	PR 74422	net7.0-windows-Release-x86-CoreCLR_release-Windows.7.Amd64.Open	console.log			runclient.py
1960114	runtime	PR 69578	net7.0-Linux-Debug-x64-mono_interpreter_release-(Debian.10.Amd64.Open)Ubuntu.1804.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-10-helix-amd64-20210304164434-56c6673	console.log	core dump		runclient.py
1959643	runtime	PR 74363	net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3	console.log	core dump		runclient.py
1959643	runtime	PR 74363	net7.0-Linux-Debug-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652	console.log	core dump		runclient.py
1959643	runtime	PR 74363	net7.0-Linux-Debug-x64-CoreCLR_release-(Centos.7.Amd64.Open)Ubuntu.1604.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-mlnet-helix-20220601183719-dde38af	console.log	core dump		runclient.py
1959643	runtime	PR 74363	net7.0-Linux-Debug-x64-CoreCLR_release-RedHat.7.Amd64.Open	console.log	core dump		runclient.py
1959643	runtime	PR 74363	net7.0-Linux-Debug-x64-CoreCLR_release-(Debian.10.Amd64.Open)Ubuntu.1804.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-10-helix-amd64-20220810215022-f344011	console.log	core dump		runclient.py
1959643	runtime	PR 74363	net7.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1959643	runtime	PR 74363	net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3	console.log	core dump		runclient.py
1959643	runtime	PR 74363	net7.0-Linux-Debug-x64-CoreCLR_release-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-amd64-20220803180115-99b3286	console.log	core dump		runclient.py
1959643	runtime	PR 74363	net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open	console.log			runclient.py
1959643	runtime	PR 74363	net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open	console.log			runclient.py
1958713	runtime	PR 74275	net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3	console.log
1958648	runtime	PR 74343	net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3	console.log
1958648	runtime	PR 74343	net7.0-Linux-Debug-arm64-Mono_release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20220427172132-97d8652	console.log
1958648	runtime	PR 74343	net7.0-Linux-Debug-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652	console.log
1958619	runtime	PR 72717	net7.0-Linux-Debug-arm64-Mono_release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20220427172132-97d8652	console.log
1958619	runtime	PR 72717	net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3	console.log
1958619	runtime	PR 72717	net7.0-Linux-Debug-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652	console.log
1958619	runtime	PR 72717	net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3	console.log
1958614	runtime	PR 74342	net7.0-Linux-Debug-arm64-Mono_release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20220427172132-97d8652	console.log
1954782	runtime	Rolling	net7.0-Linux-Release-arm64-Mono_release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20220427172132-97d8652	console.log
1954782	runtime	Rolling	net7.0-Linux-Release-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652	console.log
1954782	runtime	Rolling	net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3	console.log
1954782	runtime	Rolling	net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3	console.log
1954740	runtime	PR 73965	net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3	console.log
1954740	runtime	PR 73965	net7.0-Linux-Debug-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652	console.log
1954722	runtime	PR 74275	net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3	console.log
1954722	runtime	PR 74275	net7.0-Linux-Debug-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652	console.log
1954722	runtime	PR 74275	net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3	console.log
1954713	runtime	PR 74254	net7.0-Linux-Debug-arm64-Mono_release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20220427172132-97d8652	console.log
1954699	runtime	PR 74113	net7.0-Linux-Debug-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652	console.log
1954699	runtime	PR 74113	net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3	console.log
1954693	runtime	PR 74205	net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3	console.log
1954693	runtime	PR 74205	net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3	console.log
1954693	runtime	PR 74205	net7.0-Linux-Debug-arm64-Mono_release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20220427172132-97d8652	console.log
1954684	runtime	Rolling	net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3	console.log
1954684	runtime	Rolling	net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3	console.log
1954684	runtime	Rolling	net7.0-Linux-Release-arm64-Mono_release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20220427172132-97d8652	console.log
1954684	runtime	Rolling	net7.0-Linux-Release-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652	console.log
1954571	runtime	PR 74123	net7.0-Linux-Debug-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652	console.log
1954571	runtime	PR 74123	net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3	console.log
1946264	runtime	Rolling	net7.0-Linux-Release-x64-CoreCLR_release-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1946264	runtime	Rolling	net7.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1946224	runtime	PR 73768	net7.0-Linux-Debug-x64-Mono_release-(Centos.7.Amd64.Open)Ubuntu.1604.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-mlnet-helix-20220601183719-dde38af	console.log			runclient.py
1946013	runtime	PR 73980	net7.0-Linux-Debug-x64-Mono_release-(Centos.7.Amd64.Open)Ubuntu.1604.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-mlnet-helix-20220601183719-dde38af	console.log			runclient.py
1946003	runtime	PR 73987	net7.0-Linux-Debug-x64-Mono_release-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1945979	runtime	PR 73963	net7.0-Linux-Debug-x64-Mono_release-RedHat.7.Amd64.Open	console.log			runclient.py
1945979	runtime	PR 73963	net7.0-windows-Release-x86-CoreCLR_release-Windows.7.Amd64.Open	console.log			runclient.py
1945940	runtime	PR 67049	net7.0-windows-Debug-x64-CoreCLR_release-Windows.11.Amd64.Client.Open	console.log			runclient.py
1945820	runtime	PR 73986	net6.0-Linux-Debug-x64-Mono_release-(Centos.7.Amd64.Open)Ubuntu.1604.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-mlnet-helix-20210714125435-dde38af	console.log			runclient.py
1945786	runtime	PR 73983	net7.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1945035	runtime	PR 73967	net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3	console.log			runclient.py
1944870	runtime	PR 69527	net7.0-OSX-Debug-x64-Mono_release-OSX.1200.Amd64.Open	console.log			runclient.py
1944753	runtime	PR 73942	net7.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1942293	runtime	PR 73768	net7.0-OSX-Debug-x64-Mono_release-OSX.1200.Amd64.Open	console.log			runclient.py
1942293	runtime	PR 73768	net7.0-Linux-Debug-x64-Mono_release-(Centos.7.Amd64.Open)Ubuntu.1604.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-mlnet-helix-20220601183719-dde38af	console.log	core dump		runclient.py
1942293	runtime	PR 73768	net7.0-Linux-Debug-x64-Mono_release-RedHat.7.Amd64.Open	console.log	core dump		runclient.py
1942293	runtime	PR 73768	net7.0-Linux-Debug-x64-Mono_release-(Debian.10.Amd64.Open)Ubuntu.1804.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-10-helix-amd64-20220810215022-f344011	console.log	core dump		runclient.py
1942293	runtime	PR 73768	net7.0-Linux-Debug-x64-Mono_release-Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1941350	runtime	PR 73882	net7.0-Linux-Debug-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652	console.log	core dump		runclient.py
1941350	runtime	PR 73882	net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open	console.log			runclient.py
1941350	runtime	PR 73882	net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3	console.log	core dump		runclient.py
1941350	runtime	PR 73882	net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3	console.log	core dump		runclient.py
1941350	runtime	PR 73882	net7.0-windows-Debug-x86-CoreCLR_checked-Windows.10.Amd64.Open	console.log	core dump		runclient.py
1941350	runtime	PR 73882	net7.0-Linux-Debug-x64-CoreCLR_checked-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-amd64-20210910135833-1848e19	console.log	core dump		runclient.py
1939895	runtime	PR 73768	net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3	console.log	core dump		runclient.py
1939895	runtime	PR 73768	net7.0-Linux-Debug-x64-Mono_release-(Centos.7.Amd64.Open)Ubuntu.1604.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-mlnet-helix-20220601183719-dde38af	console.log	core dump		runclient.py
1939895	runtime	PR 73768	net7.0-Linux-Debug-x64-Mono_release-RedHat.7.Amd64.Open	console.log	core dump		runclient.py
1939895	runtime	PR 73768	net7.0-Linux-Debug-x64-Mono_release-(Debian.10.Amd64.Open)Ubuntu.1804.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-10-helix-amd64-20220810215022-f344011	console.log	core dump		runclient.py
1939895	runtime	PR 73768	net7.0-Linux-Debug-x64-Mono_release-Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1939895	runtime	PR 73768	net7.0-Linux-Debug-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652	console.log	core dump		runclient.py
1939895	runtime	PR 73768	net7.0-OSX-Debug-x64-Mono_release-OSX.1200.Amd64.Open	console.log			runclient.py
1938154	runtime	PR 73736	net7.0-windows-Release-x64-CoreCLR_checked-Windows.10.Amd64.Open	console.log	core dump		runclient.py
1938154	runtime	PR 73736	net7.0-windows-Debug-x86-CoreCLR_checked-Windows.10.Amd64.Open	console.log	core dump		runclient.py
1936905	runtime	PR 73768	net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3	console.log	core dump		runclient.py
1936905	runtime	PR 73768	net7.0-Linux-Debug-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652	console.log	core dump		runclient.py
1928330	runtime	PR 70941	net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3	console.log	core dump		runclient.py
1924484	runtime	PR 73186	net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3	console.log	core dump		runclient.py
1916944	runtime	PR 72896	net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3	console.log	core dump		runclient.py
1916354	runtime	PR 73185	net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open	console.log			runclient.py
1916354	runtime	PR 73185	net7.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1916354	runtime	PR 73185	net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3	console.log	core dump		runclient.py

Displaying 100 of 129 results

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
2	15	53

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @dotnet/area-system-collections
See info in area-owners.md if you want to be subscribed.

Issue Details

---

https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-60f3ac9cf7e44ce790/System.Collections.Concurrent.Tests/1/console.4d788105.log?%3Fhelixlogtype%3Dresult

/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Collections.Concurrent.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Collections.Concurrent.Tests (found 885 of 893 test cases)
  Starting:    System.Collections.Concurrent.Tests (parallel test collections = on, max threads = 4)
Process terminated. Assertion failed.
A QueueUserWorkItemCallback was never called!
   at System.Threading.QueueUserWorkItemCallbackBase.Finalize() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 1175
./RunTests.sh: line 168:    24 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Collections.Concurrent.Tests.runtimeconfig.json --depsfile System.Collections.Concurrent.Tests.deps.json xunit.console.dll System.Collections.Concurrent.Tests.dll 

There are several failures of this test recently, but the errors vary and it is unclear if they are related to same root cause or not.

Runfo Tracking Issue: system.collections.concurrent.tests failed in CI

Build	Definition	Kind	Run Name	Console	Core Dump	Test Results	Run Client
1909685	runtime	Rolling	net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3	console.log	core dump		runclient.py
1909465	runtime	PR 72979	net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3	console.log			runclient.py
1908206	runtime	PR 72979	net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3	console.log			runclient.py
1906378	runtime	PR 72933	net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open	console.log			runclient.py

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
2	4	4

Author:	runfoapp[bot]
Assignees:	-
Labels:	area-System.Collections, blocking-clean-ci, untriaged
Milestone:	-

[karelz]

Most of the PR failures are caused by the PRs themselves. Only 1 failure happened 3x in last week - still worth of 'blocking-clean-ci' label.

@noahfalk were the failures in your PR #72179 from 7/19 caused by your PR?

[noahfalk]

Earlier iterations of my PR did have a bug which triggered some tests to fail, as they should have. The final iteration of the PR just had the failure from #70969 which was a known issue.

[karelz]

Not blocking-clean-ci as it did not happen for 2+ weeks.