ManyConcurrentAddsTakes_ForceContentionWithToArray fails intermittently on osx-arm64

[jkotas]

Assert.DoesNotContain() Failure
Found:    0
In value: Int32[] [0]
   at System.Collections.Concurrent.Tests.ProducerConsumerCollectionTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(Double seconds) + 0x110
   at System.Collections.Concurrent!<BaseAddress>+0x9fa6bc
   at System.Reflection.DynamicInvokeInfo.Invoke(Object, IntPtr, Object[], BinderBundle, Boolean) + 0x148

 {
    "ErrorMessage" : "System.Collections.Concurrent.Tests.ProducerConsumerCollectionTests.ManyConcurrentAddsTakes_ForceContentionWithToArray",
    "BuildRetry": false
 }

Failed in #76499. Log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-76499-merge-3c4c964d671349bdbd/System.Collections.Concurrent.Tests/1/console.b1cceddc.log?helixlogtype=result

Report

Build	Definition	Test	Pull Request
74353	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)
74067	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)
73941	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)	#77934
73002	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)
72557	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)
71724	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithGetEnumerator
70399	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)	#77770
67757	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)
67272	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)
67179	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)
66666	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)	#76630
64951	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)	#77522

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	5	12

[jkotas]

Also failed in #75421 (comment)

[jkotas]

cc @VSadov @filipnavara

[filipnavara]

I believe the failure is genuine but I was not able to reliably reproduce it on my machine yet. Should we mark the test with ActiveIssue until we find the root cause?

[jkotas]

"Known Build Error" label turns on automation to monitor and collect statistics about the failure. Let's see how often it fails. If it fails too often, we are going to disable it.

[filipnavara]

It reproduces in less than a minute when running the single test in loop on M1.

[filipnavara]

In fact, it reproduces under regular CoreCLR as well:

./.dotnet/dotnet /Users/filipnavara/Projects/ConcurrentDict/bin/Debug/net7.0/ConcurrentDict.dll 
FAIL
Unhandled exception. System.Exception: Exception of type 'System.Exception' was thrown.
   at Assert.DoesNotContain(Int32 value, Int32[] array) in /Users/filipnavara/Projects/ConcurrentDict/ConcurrentDict.cs:line 92
   at BringUpTest.Test() in /Users/filipnavara/Projects/ConcurrentDict/ConcurrentDict.cs:line 50
   at BringUpTest.Main() in /Users/filipnavara/Projects/ConcurrentDict/ConcurrentDict.cs:line 18

ConcurrentDict.cs:

// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

using System;
using System.Diagnostics;
using System.Text;
using System.Collections.Concurrent;
using System.Threading.Tasks;

public class BringUpTest
{
    const int Pass = 100;
    const int Fail = -1;

    public static int Main()
    {
        for (int i = 0; i < 200; i++)
            Test();
        return Pass;
    }

    public static void Test()
    {
        IProducerConsumerCollection<int> c = new ConcurrentQueue<int>();
        const int MaxCount = 4;

        DateTime end = DateTime.UtcNow + TimeSpan.FromSeconds(1);

        Task addsTakes = Task.Factory.StartNew(() =>
        {
            while (DateTime.UtcNow < end)
            {
                for (int i = 1; i <= MaxCount; i++)
                {
                    Assert.True(c.TryAdd(i));
                }
                for (int i = 1; i <= MaxCount; i++)
                {
                    int item;
                    Assert.True(c.TryTake(out item));
                    Assert.InRange(item, 1, MaxCount);
                }
            }
        });

        while (DateTime.UtcNow < end)
        {
            int[] arr = c.ToArray();
            Assert.InRange(arr.Length, 0, MaxCount);
            Assert.DoesNotContain(0, arr); // make sure we didn't get default(T)
        }

        addsTakes.GetAwaiter().GetResult();
        Assert.Equal(0, c.Count);
    }
}

class Assert
{
    public static void Equal(int expected, int actual)
    {
        if (expected != actual)
        {
            Console.WriteLine($"{actual} != {expected}");
            throw new Exception();
        }
    }

    public static void True(bool actual)
    {
        if (!actual)
        {
            Console.WriteLine($"!{actual}");
            throw new Exception();
        }
    }

    public static void InRange(int actual, int min, int max)
    {
        if (actual < min || actual > max)
        {
            Console.WriteLine($"{actual} not in {min}-{max}");
            throw new Exception();
        }
    }

    public static void DoesNotContain(int value, int[] array)
    {
        if (Array.Exists(array, e => e == value))
        {
            Console.WriteLine($"FAIL");
            throw new Exception();
        }
    }
}