[RISC-V] Test build doesn't make crossgen2 executable in RISC-V.

[clamp03]

Recently, after test build, I cannot find crossgen2 executable.

How I build test.

$ export BuildAllTestsAsStandalone=true
$ ROOTFS_DIR=<riscv64 rootfs> ./src/tests/build.sh -riscv64 -Release -priority1 -cross -p:UseLocalAppHostPack=true

Files in crossgen2 directory of old artifact.

$ ls artifacts/tests/coreclr/linux.riscv64.Release/Tests/Core_Root/crossgen2
corerun       libclrjit_unix_riscv64_riscv64.so  libjitinterface_riscv64.so         libSystem.IO.Compression.Native.a   libSystem.Native.so                                LICENSE.TXT
createdump    libcoreclr.so                      libmscordaccore.so                 libSystem.IO.Compression.Native.so  libSystem.Net.Security.Native.a
crossgen2     libcoreclrtraceptprovider.so       libmscordbi.so                     libSystem.IO.Ports.Native.a         libSystem.Net.Security.Native.so
libclrgc.so   libhostfxr.so                      libSystem.Globalization.Native.a   libSystem.IO.Ports.Native.so        libSystem.Security.Cryptography.Native.OpenSsl.a
libclrjit.so  libhostpolicy.so                   libSystem.Globalization.Native.so  libSystem.Native.a                  libSystem.Security.Cryptography.Native.OpenSsl.so

Files in crossgen2 directory of latest artifact.

$ ls artifacts/tests/coreclr/linux.riscv64.Release/Tests/Core_Root/crossgen2
libclrjit_unix_riscv64_x64.so  libjitinterface_x64.so

I cannot find crossgen2_publish directory in artifacts/bin/

From #106965, we need crossgen2 executable to execute some R2R tests in clrtest.
These are failed clrtests due to this issue on RISC-V.

Regressions/coreclr/GitHub_49826/test49826/test49826.sh                                                                                                                                                                                       
Regressions/coreclr/GitHub_49982/test49982/test49982.sh                                                                                                                                                                                       
Regressions/coreclr/GitHub_61104/test61104/test61104.sh                                                                                                                                                                                       
baseservices/TieredCompilation/BasicTestWithMcj/BasicTestWithMcj.sh                                                                                                                                                                           
baseservices/TieredCompilation/BasicTest_DefaultMode_R2r/BasicTest_DefaultMode_R2r.sh                                                                                                                                                         
baseservices/TieredCompilation/BasicTest_QuickJitForLoopsOff_R2r/BasicTest_QuickJitForLoopsOff_R2r.sh                                                                                                                                         
baseservices/TieredCompilation/BasicTest_QuickJitForLoopsOn_R2r/BasicTest_QuickJitForLoopsOn_R2r.sh                                                                                                                                           
baseservices/TieredCompilation/BasicTest_QuickJitOn_R2r/BasicTest_QuickJitOn_R2r.sh                                                                                                                                                           
baseservices/TieredCompilation/BasicTest_QuickJitOff_R2r/BasicTest_QuickJitOff_R2r.sh                                                                                                                                                         
readytorun/GenericCycleDetection/Breadth1Test/Breadth1Test.sh                                                                                                                                                                                 
readytorun/GenericCycleDetection/Depth1Test/Depth1Test.sh                                                                                                                                                                                     
readytorun/GenericCycleDetection/Depth3Test/Depth3Test.sh                                                                                                                                                                                     
readytorun/crossgen2/crossgen2smoke/crossgen2smoke.sh                                                                                                                                                                                         
readytorun/determinism/crossgen2determinism/crossgen2determinism.sh                                                                                                                                                                           
readytorun/fieldlayout/fieldlayout/fieldlayout.sh                                                                                                                                                                                             
readytorun/tests/mainv1/mainv1.sh                                                                                                                                                                                                             
readytorun/tests/mainv2/mainv2.sh                                                                                                                                                                                                             
readytorun/tests/genericsload/callgenericctor/callgenericctor.sh                                                                                                                                                                              
readytorun/tests/genericsload/usegenericfield/usegenericfield.sh                                                                                                                                                                              
readytorun/coreroot_determinism/coreroot_determinism/coreroot_determinism.sh      

Error message is like ./test49826.sh: line 328: /linux.riscv64.Checked/Tests/Core_Root/crossgen2/crossgen2: No such file or directory message.

As far as I know, test build fails from #107772 and test build doesn't make crossgen2 executable from #108693

I tried some to make crossgen2. However, I can't.
(*. I can just made a crosgen2 and passed tests if I revert #107772, #108693, and #103375 and set PublishReadyToRun to false in crossgen2_publish.csproj. I never think it is a right approach.)
I don't know well about build environment. Could you please help?
Thank you.

cc @dotnet/samsung @jkotas @am11

[jkotas]

The solution for this is discussed in #105004 (#105004 (comment) is outline of the solution).

[am11]

As far as I know, test build fails from #107772

Problem is that PR tied crossgen2_publishing with NativeAotSupported=true (and linux-riscv64 has it false), while the project itself accounts for AotOrSingleFile=true. I am not sure if deleting that part of the condition from this line

runtime/eng/Subsets.props

Line 361 in 830ce3a

<ProjectToBuild Include="$(CoreClrProjectRoot)tools\aot\crossgen2\crossgen2_publish.csproj" Condition="'$(NativeAotSupported)' == 'true'" Category="clr" />

will make any difference or give other errors, but you can try.

It has gotten even worse for platforms where target OS is non-linux, and freebsd has NativeAotSupported=true while illumos has it false. I haven't figured out the all-encompassing magic spell yet. Tried multiple different things, but the build is quite complex..

[jkotas]

Tried multiple different things, but the build is quite complex..

Feel free to share what you have tried, so that we can iterated on it.

[am11]

@jkotas, I was force pushing to the other branch main...am11:runtime:feature/freebsd-port/outputrid2. Here are the freebsd logs https://github.com/am11/CrossRepoCITesting/actions/workflows/freebsd-x64-runtime-build.yml

so far I've gotten to the point where restore succeeds with -p:StageOneBuild=true. Currently I'm disabling projects looking for targetting packs with StageOneBuild != true one by one. This is the challenging part because despite illink projects being disabled in subsets, livebuild targets require it to be built, and then it runs into the same The shared framework must be built before the local targeting pack can be consumed. order dependent issue.. Disabling it from livebuild makes SDK unhappy: IsAotCompatible and EnableAotAnalyzer are not supported for the target framework. Consider multi-targeting to a supported framework to enable ahead-of-time compilation analysis, and set IsAotCompatible only for the supported frameworks.

Locally I'm on arm64 machine and build-rootfs for freebsd-arm64 wasn't working (and x64 emulation doesn't work with dotnet), so I'm using github workflow. I will switch to x64 machine on the weekend to reduce the iteration time. Meanwhile, if someone has ideas how to improve, feel free. 😅

[clamp03]

@jkotas @am11 Thank you!

will make any difference or give other errors, but you can try.

I tried. It built without any error. However, crossgen2 executable is not made. I can't find differences.

main...am11:runtime:feature/freebsd-port/outputrid2

With -p:StageOneBuild=true, it fails.

/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj : error NU1101: Unable to find package Microsoft.NETCore.App.Runtime.linux-riscv64. No packages exist with this id in source(s): dotnet-eng, dotnet-libraries, dotnet-libraries-transport, dotnet-public, dotnet-tools, dotnet10, dotnet10-transport, dotnet9, dotnet9-transport [/runtime/Build.proj]                                              
/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj : error NU1101: Unable to find package Microsoft.NETCore.App.Host.linux-riscv64. No packages exist with this id in source(s): dotnet-eng, dotnet-libraries, dotnet-libraries-transport, dotnet-public, dotnet-tools, dotnet10, dotnet10-transport, dotnet9, dotnet9-transport [/runtime/Build.proj]                                                 
/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj : error NU1101: Unable to find package Microsoft.AspNetCore.App.Runtime.linux-riscv64. No packages exist with this id in source(s): dotnet-eng, dotnet-libraries, dotnet-libraries-transport, dotnet-public, dotnet-tools, dotnet10, dotnet10-transport, dotnet9, dotnet9-transport [/runtime/Build.proj] 

[am11]

With -p:StageOneBuild=true, it fails.

It is work in progress and I'm focused on freebsd-x64, which is one part of the matrix (that targetos != hostos added in toolAot.targets can be relaxed for any non linux-arm/64 crossbuild). Right now it clears restore but tries to build stuff which depend on LKG apphost:
https://github.com/am11/CrossRepoCITesting/actions/runs/11389279815/job/31688193176

  Restored /runtime/src/coreclr/runtime-prereqs.proj (in 1.53 sec).
  ILLink.RoslynAnalyzer -> /runtime/artifacts/bin/ILLink.RoslynAnalyzer/Debug/netstandard2.0/ILLink.RoslynAnalyzer.dll
  ILLink.CodeFixProvider -> /runtime/artifacts/bin/ILLink.CodeFixProvider/Debug/netstandard2.0/ILLink.CodeFixProvider.dll
/runtime/eng/targetingpacks.targets(116,5): error : The shared framework must be built before the local targeting pack can be consumed. [/runtime/src/tools/illink/src/linker/ref/Mono.Linker.csproj]
##vso[task.logissue type=error;sourcepath=/runtime/eng/targetingpacks.targets;linenumber=116;columnnumber=5;code=;](NETCORE_ENGINEERING_TELEMETRY=Build) The shared framework must be built before the local targeting pack can be consumed.

although all illink projects are skipped in subsets.props, this one gets pulled by livebuidls.

When we started #105004, there was no issue on linux-<other-arch> type of crossbuild, and the only issue on <other-os>-arch crossbuild was, at the end of successful build it was getting linux ilc binary in artifacts/packages. Now the infra is changed, PR is solving other problems to get back to that 'single' problem state where we originally were. 🥲

[am11]

@shushanhf @sunlijun-610 are you getting similar issues when building crossgen2 or nativeatot on loongarch with latest main branch? My guess is it should be failing on LA64 same way it does on other community platforms.

[shushanhf]

@shushanhf @sunlijun-610 are you getting similar issues when building crossgen2 or nativeatot on loongarch with latest main branch? My guess is it should be failing on LA64 same way it does on other community platforms.

In fact, we always build and test the runtime and SDK on LoongArch64-PC directly and the crossgen2 is OK.
If building the runtime by cross mode, we don't do it. We will try to build it by cross mode.

[am11]

In fact, we always build and test the runtime and SDK on LoongArch64-PC directly and the crossgen2 is OK. If building the runtime by cross mode, we don't do it. We will try to build it by cross mode.

@shushanhf, thanks for the info. If we have the previous SDK in runtime/.dotnet, which matches

runtime/global.json

Line 3 in 3aa1ec5

"version": "9.0.100",

(which will soon be updated to 10.0.0-alpha1 when #110477 is merged), the runtime/build.sh should work in theory. But in practice, since #107772 is merged, it requires ILCompiler nupkg to publish crossgen2 among other things when building on the device (HostArch==TargetArch)

runtime/eng/Subsets.props

Lines 125 to 127 in 3aa1ec5

	<_NativeAotSupportedArch Condition="'$(TargetArchitecture)' == 'x64' or '$(TargetArchitecture)' == 'arm64' or '$(TargetArchitecture)' == 'arm' or '$(TargetArchitecture)' == 'loongarch64' or ('$(TargetOS)' == 'windows' and '$(TargetArchitecture)' == 'x86')">true</_NativeAotSupportedArch>
	<NativeAotSupported Condition="'$(_NativeAotSupportedOS)' == 'true' and '$(_NativeAotSupportedArch)' == 'true'">true</NativeAotSupported>
	<UseNativeAotForComponents Condition="'$(NativeAotSupported)' == 'true' and '$(TargetOS)' == '$(HostOS)' and '$(TargetsLinuxBionic)' != 'true'">true</UseNativeAotForComponents>

I was trying to understand what is your workaround for this part on current main branch?