Mono JIT/HardwareIntrinsics - Massive failures

[karelz]

63x failures in each run. Recent regression as of 9/13.

Platform: mono Linux x64 Release @ Ubuntu.1804.Amd64.Open

Last 30 days - JIT.HardwareIntrinsics.X86.Ssse3 in Runfo as of 9/14

• First failure in PR 14818 on 9/13
• First rolling run affected: 15289

There are more JIT.HardwareIntrinsics.* Work Items affected.

Example of error in Console log:

    JIT/HardwareIntrinsics/X86/Ssse3/Ssse3_r/Ssse3_r.sh [FAIL]
      
      Return code:      1
      Raw output file:      /datadisks/disk1/work/B0980979/w/B4440A48/uploads/Reports/JIT.HardwareIntrinsics/X86/Ssse3/Ssse3_r/Ssse3_r.output.txt
      Raw output:
      BEGIN EXECUTION
      /datadisks/disk1/work/B0980979/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Ssse3_r.dll ''
      Supported ISAs:
        AES:       True
        AVX:       False
        AVX2:      False
        AVXVNNI:   False
        BMI1:      True
        BMI2:      True
        FMA:       False
        LZCNT:     True
        PCLMULQDQ: True
        POPCNT:    True
        SSE:       True
        SSE2:      True
        SSE3:      True
        SSE4.1:    True
        SSE4.2:    True
        SSSE3:     True
        X86Serialize: False
      
      Beginning test case Abs.Byte at 9/13/2022 10:36:21 PM
      Random seed: 20010415; set environment variable CORECLR_SEED to this value to repro
      
      Beginning scenario: RunBasicScenario_UnsafeRead
      Beginning scenario: RunBasicScenario_Load
      Beginning scenario: RunBasicScenario_LoadAligned
      Beginning scenario: RunReflectionScenario_UnsafeRead
      ERROR!!!-System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

[karelz]

This is significantly blocking CI (19x impacted runs in ~1 day). We need to roll back the change that caused it or disable the test ASAP.

@SamMonoRT @BrzVlad @fanyang-mono can you please help route it?

[karelz]

BTW: I think I saw in one of the logs inner exception to be PlatformNotSupportedException, but I can't find it now.

[SamMonoRT]

@tannergooding @fanyang-mono - can you confirm these failures are not related to #75470

[tannergooding]

@SamMonoRT, Int128 has no relation to the HardwareIntrinsics tests. They aren't used by eachother in any way.

The Mono llvmaot failures look to be an issue with the RunReflectionScenario and is likely something to do with all the new SIMD bringup Mono has been doing.

I'd guess there isn't handling for the recursive pattern that the intrinsics currently follow.

[tannergooding]

It might be related to #75438 or one of the other recent Mono SIMD changes

[danmoseley]

It seems from above that these run on PR validation? So unless there's flakiness, it should be possible to figure which PR first merged with these failures?

[fanyang-mono]

Right, I am working on it now.

[BrzVlad]

Here is the timeline of some issues related to simd intrinsics.

There were some failures on android which I investigated, #74797 was a fix for it by disabling intrinsics. There were no llvm failures noticed on that PR, merged on main, roughly one week ago. Then tried to backport to .net 7. Noticed these same failures here on the backport PR. Tested locally my changed, saw that my change was introducing failures that were awkward to fix so gave up on the backport.

Yesterday I disabled that original change via #75438, so behavior is as before. I tested again locally to see the impact of my revert. I notice again that the revert fixes the intrinsics issues for me so I don't understand why there are still failures today on CI for this. Since my PR didn't have failures but the backport PR did, I don't fully trust what is going on.

Note that locally I didn't run the XUnit wrapper, I just ran runtime tests one at a time while aot-ing with llvm SPC.dll and System.Runtime.Intrinsics.dll