[6.0] JIT.HardwareIntrinsics test failures in mono arm64 linux llvmaot

[carlossanlop]

Found in the internal->public merge PR for 6.0: #83404

• Queue: Mono llvmaot Pri0 Runtime Tests Run Linux arm64 release
• Job: https://dev.azure.com/dnceng-public/public/_build/results?buildId=204115&view=logs&j=74e689b1-5e71-5f30-9486-7c2c5ff44021&t=bc7664d5-2e7c-57dc-1337-ab0ada348ecf
• Log file: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-83404-merge-2c2fd1a68a554032ad/JIT.HardwareIntrinsics/1/console.5bc1a453.log?helixlogtype=result
• Some examples of callstacks:

    JIT/HardwareIntrinsics/Arm/Dp/Dp_r/Dp_r.sh [FAIL]
      
      Return code:      1
      Raw output file:      /root/helix/work/workitem/uploads/Reports/JIT.HardwareIntrinsics/Arm/Dp/Dp_r/Dp_r.output.txt
      Raw output:
      BEGIN EXECUTION
      /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Dp_r.dll ''
      Supported ISAs:
        AdvSimd:   True
        Aes:       True
        ArmBase:   True
        Crc32:     True
        Dp:        True
        Rdm:       True
        Sha1:      True
        Sha256:    True

      Ending test case at 3/14/2023 6:17:17 PM
      Expected: 100
      Actual: 0
      END EXECUTION - FAILED
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/root/helix/work/correlation
      > /root/helix/work/workitem/e/JIT/HardwareIntrinsics/Arm/Dp/Dp_r/Dp_r.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at JIT_HardwareIntrinsics._Arm_Dp_Dp_r_Dp_r_._Arm_Dp_Dp_r_Dp_r_sh()
      Output:
        
        Return code:      1
        Raw output file:      /root/helix/work/workitem/uploads/Reports/JIT.HardwareIntrinsics/Arm/Dp/Dp_r/Dp_r.output.txt
        Raw output:
        BEGIN EXECUTION
        /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Dp_r.dll ''
        Supported ISAs:
          AdvSimd:   True
          Aes:       True
          ArmBase:   True
          Crc32:     True
          Dp:        True
          Rdm:       True
          Sha1:      True
          Sha256:    True

        Ending test case at 3/14/2023 6:17:17 PM
        Expected: 100
        Actual: 0
        END EXECUTION - FAILED
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=/root/helix/work/correlation
        > /root/helix/work/workitem/e/JIT/HardwareIntrinsics/Arm/Dp/Dp_r/Dp_r.sh
    JIT/HardwareIntrinsics/Arm/Rdm.Arm64/Rdm.Arm64_ro/Rdm.Arm64_ro.sh [FAIL]
      
      Return code:      1
      Raw output file:      /root/helix/work/workitem/uploads/Reports/JIT.HardwareIntrinsics/Arm/Rdm.Arm64/Rdm.Arm64_ro/Rdm.Arm64_ro.output.txt
      Raw output:
      BEGIN EXECUTION
      /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Rdm.Arm64_ro.dll ''
      Supported ISAs:
        AdvSimd:   True
        Aes:       True
        ArmBase:   True
        Crc32:     True
        Dp:        True
        Rdm:       True
        Sha1:      True
        Sha256:    True

        Ending test case at 3/14/2023 6:18:27 PM
        Expected: 100
        Actual: 0
        END EXECUTION - FAILED
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=/root/helix/work/correlation
        > /root/helix/work/workitem/e/JIT/HardwareIntrinsics/Arm/Rdm/Rdm_ro/Rdm_ro.sh
    JIT/HardwareIntrinsics/Arm/Rdm.Arm64/Rdm.Arm64_r/Rdm.Arm64_r.sh [FAIL]
      
      Return code:      1
      Raw output file:      /root/helix/work/workitem/uploads/Reports/JIT.HardwareIntrinsics/Arm/Rdm.Arm64/Rdm.Arm64_r/Rdm.Arm64_r.output.txt
      Raw output:
      BEGIN EXECUTION
      /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Rdm.Arm64_r.dll ''
      Supported ISAs:
        AdvSimd:   True
        Aes:       True
        ArmBase:   True
        Crc32:     True
        Dp:        True
        Rdm:       True
        Sha1:      True
        Sha256:    True

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[steveisok]

/cc @SamMonoRT

[SamMonoRT]

@carlossanlop - how do I query for list of backported PRs approved in the timespan just before these started failing ?

[carlossanlop]

Hmm good question...

I think the best option is for manually write a Kusto query with your desired filters: https://kusto.azure.com/clusters/engsrvprod/databases/engineeringdata

I would alternatively try by using the .NET Core Telemetry Report (public).

• Right click on the runtime horizontal bar.
• Select "Drill through" -> "Build Investigation" in the context menu.
• You'll be shown the execution history of the rolling builds (for main).

From there, it's just a matter of finding a way to filter by:

• Servicing branch instead of main
• Increase the timespan to show older results
• Figure out if you can also filter by keywords

@steveisok do you have a better option?

[steveisok]

@steveisok do you have a better option?

I don't other than bisecting, but that would be painful.