Re-enable arm64 and x64 Mono fullAOT llvm and mini jobs

[SamMonoRT]

Description

Currently, the Mono fullAOT LLVM jobs are encountering failures and don't provide any test coverage. Specifically, the linux-arm64 job must build the cross-compiler using an image which incorporates both x64 and arm64 rootfs. Additionally, the aot-compiler.c utilizes GNU as which is not available since we don't have GNU binutils on the Mariner.

To resolve this issue, we have to fix the mono-aot-cross build for linux-arm64 to use target rootfs when building the runtime and host rootfs when building the compiler. After that, the aot-compiler.c has to be updated to use llvm-as or clang instead of GNU as.

Additionally, the AOT compiler with llvm 16 toolchain hits OOM issue with CBL_Mariner images.

Tasks

• [mono][infra] Fix mono-aot-cross build for linux-arm64 using biarch image #91019
• [infra] Enable linux-x64 Mono fullAOT mini job on extra platforms #96332
• [mono][infra] Enable runtime tests in fullAOT LLVM mode on linux-x64 #92057
• [mono][aot] Investigate RAM consumption in Mono AOT compiler #95791
• [WIP][mono] Enable runtime tests in AOT modes on linux-arm64 #100685
• Add macOS fullAOT with interpreter fallback runtime test coverage
• Backport changes to .NET 8

Initial discussion can be found at #86324 (comment).

Disabled tests

Upon the initial setup of the jobs, the following tests require triage.

FullAOT mini

AOT compilation failure:

src/tests/JIT/Methodical/Boxing/boxunbox/BoxPatternMatchAndSideEffects.csproj
src/tests/JIT/Methodical/Methodical_others.csproj
src/tests/JIT/Regression/JitBlue/GitHub_26491/GitHub_26491.ilproj
src/tests/JIT/Regression/Regression_3.csproj
src/tests/Interop/StringMarshalling/AnsiBSTR/AnsiBStrTest.csproj
src/tests/Interop/StringMarshalling/BSTR/BSTRTest.csproj
src/tests/Interop/StringMarshalling/LPTSTR/LPTSTRTest.csproj
src/tests/Interop/StringMarshalling/VBByRefStr/VBByRefStrTest.csproj
src/tests/Interop/Interop.csproj
src/tests/readytorun/coreroot_determinism/readytorun_coreroot_determinism.csproj
src/tests/readytorun/readytorun.csproj

Execution failure:

JIT/Methodical/Invoke/25params/25paramMixed_d/**
JIT/Methodical/Invoke/SEH/catchfault_jmp_d/**
JIT/Methodical/Invoke/SEH/catchfinally_jmpind_d/**
JIT/Methodical/Invoke/SEH/catchfinally_jmp_d/**
JIT/Methodical/tailcall/deep_array_d/**
JIT/Methodical/tailcall/deep_array_nz_d/**
JIT/Methodical/tailcall/deep_gc_d/**
JIT/Methodical/tailcall/deep_inst_d/**
JIT/Methodical/tailcall/deep_value_d/**
JIT/Methodical/tailcall/deep_virt_d/**
JIT/Methodical/VT/callconv/jumps2_d/**
JIT/Directed/zeroinit/tail_zeroinit/**
JIT/Directed/callconv/ThisCall/ThisCallTest/**
JIT/Directed/pinvoke/tail_pinvoke/**
JIT/Methodical/Invoke/25params/25paramMixed_r/**
JIT/Methodical/Invoke/SEH/catchfault_jmp_r/**
JIT/Methodical/Invoke/SEH/catchfinally_jmpind_r/**
JIT/Methodical/Invoke/SEH/catchfinally_jmp_r/**
JIT/Methodical/tailcall/deep_array_nz_r/**
JIT/Methodical/tailcall/deep_array_r/**
JIT/Methodical/tailcall/deep_gc_r/**
JIT/Methodical/tailcall/deep_inst_r/**
JIT/Methodical/tailcall/deep_value_r/**
JIT/Methodical/tailcall/deep_virt_r/**
JIT/Methodical/VT/callconv/jumper4_r/**
JIT/Methodical/VT/callconv/jumper5_r/**
JIT/Methodical/VT/callconv/jumps2_r/**
Regressions/coreclr/16354/notimplemented/**
JIT/Directed/perffix/primitivevt/callconv3_d/**
JIT/Directed/perffix/primitivevt/callconv3_r/**
JIT/SIMD/Vector3Interop_r/Vector3Interop_r.**
JIT/SIMD/Vector3Interop_ro/Vector3Interop_ro.**
JIT/Regression/CLR-x86-JIT/V1.2-M01/b13452/b13452/**
JIT/Regression/CLR-x86-JIT/V2.0-Beta2/b353858/b353858/**
JIT/Methodical/Boxing/misc/tailjump_r/**
JIT/Methodical/Boxing/misc/tailjump_d/**
baseservices/compilerservices/UnsafeAccessors/UnsafeAccessorsTests/**
baseservices/TieredCompilation/TieredVtableMethodTests/**
baseservices/varargs/varargsupport/**
JIT/opt/Devirtualization/Comparer_get_Default/Comparer_get_Default.**
JIT/Regression/JitBlue/Runtime_80731/Runtime_80731/**
Loader/classloader/DictionaryExpansion/DictionaryExpansion/**
Loader/classloader/explicitlayout/NestedStructs/case03/**
Loader/classloader/explicitlayout/NestedStructs/case04/**
Loader/classloader/explicitlayout/NestedStructs/case05/**
Loader/classloader/explicitlayout/Regressions/ASURT/ASURT150271/test13/**
Loader/classloader/MethodImpl/generics_override1/**
Loader/classloader/Statics/Misc/LiteralStatic/**
Loader/CustomAttributes/DynamicObjects/**
JIT/jit64/localloc/call/call05_dynamic/**
JIT/jit64/verif/sniff/fg/ver_fg_13/**
JIT/Directed/coverage/importer/badendfinally/**
JIT/Directed/coverage/importer/Desktop/badendfinally_d/**
JIT/Directed/coverage/importer/Desktop/badendfinally_r/**
JIT/Directed/coverage/importer/ceeillegal/**
JIT/Directed/coverage/importer/Desktop/ceeillegal_d/**
JIT/Directed/coverage/importer/Desktop/ceeillegal_r/**
JIT/SIMD/Vector3Interop_r/**
JIT/SIMD/Vector3Interop_ro/**
JIT/opt/Devirtualization/Comparer_get_Default/**
baseservices/varargs/varargsupport_r/**
Loader/classloader/MethodImpl/CovariantReturns/ReturnTypeValidation/**
Loader/classloader/MethodImpl/CovariantReturns/UnitTest/UnitTest_GVM/**
Interop/StringMarshalling/AnsiBSTR/AnsiBStrTest/**
Interop/StringMarshalling/BSTR/BSTRTest/**
Interop/StringMarshalling/LPTSTR/LPTSTRTest/**
Interop/StringMarshalling/VBByRefStr/VBByRefStrTest/**
JIT/Methodical/Boxing/boxunbox/BoxPatternMatchAndSideEffects/**
JIT/Intrinsics/TypeIntrinsics_r/**
JIT/Intrinsics/TypeIntrinsics_ro/**
ilasm/System/Runtime/CompilerServices/MethodImplOptionsTests/**
ilasm/PortablePdb/IlasmPortablePdbTests/**

FullAOT LLVM

AOT compilation failure:

src/tests/JIT/Methodical/Boxing/boxunbox/BoxPatternMatchAndSideEffects.csproj
src/tests/JIT/Methodical/Methodical_others.csproj
src/tests/JIT/Regression/JitBlue/GitHub_26491/GitHub_26491.ilproj
src/tests/JIT/Regression/Regression_3.csproj
src/tests/Interop/StringMarshalling/AnsiBSTR/AnsiBStrTest.csproj
src/tests/Interop/StringMarshalling/BSTR/BSTRTest.csproj
src/tests/Interop/StringMarshalling/LPTSTR/LPTSTRTest.csproj
src/tests/Interop/StringMarshalling/VBByRefStr/VBByRefStrTest.csproj
src/tests/Interop/Interop.csproj
src/tests/readytorun/coreroot_determinism/readytorun_coreroot_determinism.csproj
src/tests/readytorun/readytorun.csproj

Execution failure:

JIT/Methodical/Invoke/25params/25paramMixed_r
JIT/Methodical/Invoke/SEH/catchfault_jmp_r
JIT/Methodical/Invoke/SEH/catchfinally_jmpind_r
JIT/Methodical/Invoke/SEH/catchfinally_jmp_r
JIT/Methodical/tailcall/deep_array_nz_r
JIT/Methodical/tailcall/deep_array_r
JIT/Methodical/tailcall/deep_gc_r
JIT/Methodical/tailcall/deep_inst_r
JIT/Methodical/tailcall/deep_value_r
JIT/Methodical/tailcall/deep_virt_r
JIT/Methodical/VT/callconv/jumper4_r
JIT/Methodical/VT/callconv/jumper5_r
JIT/Methodical/VT/callconv/jumps2_r
JIT/Directed/perffix/primitivevt/callconv3_d
JIT/Directed/perffix/primitivevt/callconv3_r
JIT/jit64/localloc/call/call05_dynamic
JIT/jit64/verif/sniff/fg/ver_fg_13
JIT/Methodical/Invoke/25params/25paramMixed_d
JIT/Methodical/Invoke/SEH/catchfault_jmp_d
JIT/Methodical/Invoke/SEH/catchfinally_jmpind_d
JIT/Methodical/Invoke/SEH/catchfinally_jmp_d
JIT/Methodical/tailcall/deep_array_d
JIT/Methodical/tailcall/deep_array_nz_d
JIT/Methodical/tailcall/deep_gc_d
JIT/Methodical/tailcall/deep_inst_d
JIT/Methodical/tailcall/deep_value_d
JIT/Methodical/tailcall/deep_virt_d
JIT/Methodical/VT/callconv/jumps2_d
JIT/SIMD/Vector3Interop_r
JIT/SIMD/Vector3Interop_ro
JIT/HardwareIntrinsics/X86/Sse2.X64/StoreNonTemporal_r
JIT/HardwareIntrinsics/X86/Sse2.X64/StoreNonTemporal_ro
JIT/Directed/coverage/importer/badendfinally
JIT/Directed/coverage/importer/Desktop/badendfinally_d
JIT/Directed/coverage/importer/Desktop/badendfinally_r
JIT/Directed/coverage/importer/ceeillegal
JIT/Directed/coverage/importer/Desktop/ceeillegal_d
JIT/Directed/coverage/importer/Desktop/ceeillegal_r
JIT/HardwareIntrinsics/General/Vector128_1
baseservices/varargs/varargsupport_r
JIT/Methodical/Boxing/misc/tailjump_r
Regressions/coreclr/16354/notimplemented
JIT/Regression/CLR-x86-JIT/V1.2-M01/b13452/b13452
JIT/Regression/CLR-x86-JIT/V2.0-Beta2/b353858/b353858
JIT/Methodical/Boxing/misc/tailjump_d
JIT/Directed/zeroinit/tail_zeroinit
JIT/Directed/callconv/ThisCall/ThisCallTest
JIT/Directed/pinvoke/tail_pinvoke
Loader/classloader/generics/ByRefLike/ValidateNegative
JIT/Regression/JitBlue/Runtime_80731/Runtime_80731
JIT/opt/Devirtualization/Comparer_get_Default
Loader/classloader/DictionaryExpansion/DictionaryExpansion
Loader/classloader/explicitlayout/NestedStructs/case03
Loader/classloader/explicitlayout/NestedStructs/case04
Loader/classloader/explicitlayout/NestedStructs/case05
Loader/classloader/explicitlayout/Regressions/ASURT/ASURT150271/test13
Loader/classloader/MethodImpl/CovariantReturns/ReturnTypeValidation/ImplicitOverrideSameSigAsDecl
Loader/classloader/MethodImpl/CovariantReturns/ReturnTypeValidation/OverrideSameSigAsDecl
Loader/classloader/MethodImpl/CovariantReturns/UnitTest/UnitTest_GVM
Loader/classloader/MethodImpl/generics_override1
Loader/classloader/Statics/Misc/LiteralStatic
Loader/CustomAttributes/DynamicObjects
baseservices/compilerservices/UnsafeAccessors/UnsafeAccessorsTests
baseservices/TieredCompilation/TieredVtableMethodTests
baseservices/varargs/varargsupport

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Refer to #86324 (comment), to re-enable the disabled lane.

Author:	SamMonoRT
Assignees:	directhex, kotlarmilos
Labels:	area-Infrastructure-mono
Milestone:	9.0.0

[kotlarmilos]

Mono fullAOT linux runtime tests now utilize the new Mariner image. This image does not include binutils, and as a result, GNU as cannot be used as a native assembler. The AOT compiler utilizes llvm opt and ilc to generate the llvm binary file. Then, a native assembler and linker are required to produce the final binary.

One option is to utilize llvm-mc (clang), which can convert assembly into a binary object file. However, I am not sure if there's a 1:1 mapping between the llvm-mc and GNU as. Another option is to ship binutils with the image.

For linux-arm64, either option would require a cross-compiled toolchain for x64 targeting arm64.

/cc: @directhex @lambdageek

[kotlarmilos]

After several attempts to enable fullAOT llvm runtime tests on the extra-platforms, the job consistently hits OOM issue after 2h. To simplify the process, the job compiles only the baseservices/mono and System.Private.CoreLib with the maxcpucount set to 1.

The CI is using an azure instance with 16gb, docker on top of that is limited to less, probably 8gb. At peak, the AOT compilation takes about 16gb to AOT compile the System.Private.CoreLib. When I tried to reproduce the issue locally on the cbl-mariner-2.0-cross-amd64, it completed the compilation successfully within 10min. There is a tracking issue in azdo where we have requested a hardware upgrade dotnet/dnceng#1163.

@steveisok @SamMonoRT While we work on resolving the OOM issue, I would appreciate hearing your thoughts on temporarily disabling the affected jobs to ensure a consistently green CI.

[steveisok]

I'm fine with disabling for the time being.

I wonder if requiring so much memory is a sign we can improve our aot compiler utilization?

[kotlarmilos]

I wonder if requiring so much memory is a sign we can improve our aot compiler utilization?

Yes, let's create a tracking issue so that we can brainstorm our next steps. Additionally, I'll temporarily disable the failing jobs.