Cross built ILCompiler NuGet contains HostOS (Linux) ELFs not TargetOS (FreeBSD) ELFs

[Thefrank]

Overview:

When cross compiling runtime, the ILCompiler (and NuGet) that is produced contains HostOS (Linux) and not TargetOS (FreeBSD) ELFs.

Reproduction:

Using a docker container on Linux pulled from: https://raw.githubusercontent.com/dotnet/versions/master/build-info/docker/image-info.dotnet-dotnet-buildtools-prereqs-docker-main.json containing a FreeBSD ROOTFS and the most recent net9 preview tag from runtime: v9.0.0-preview.5.24306.7

docker run -e ROOTFS_DIR=/crossrootfs/$ROOTFSARCH -v ${BUILD_SOURCESDIRECTORY}/runtime:/runtime $DOTNETDOCKERCONTAINERUSED /runtime/build.sh -c ${{ parameters.buildType }} -cross -os freebsd -arch $ROOTFSARCH -ci /p:OfficialBuildId=$OFFICIALBUILDID --subset clr+mono+mono.manifests+tools+libs+host+packs

Expected behavior:

ELFs should be like other items generated for TargetOS:

$file crossgen2 
crossgen2: ELF 64-bit LSB pie executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 13.2, FreeBSD-style, BuildID[sha1]=54a7f1c2a4752435c2cffd15eeb959f609966907, stripped

Actual behavior:

The resulting runtime.freebsd-x64.Microsoft.DotNet.ILCompiler.9.0.0-preview.5.24306.7.nupkg contains Linux ELFs and FreeBSD libs.

$find ./ * | xargs file
./:                                           directory
./ILCompiler.RyuJit.pdb:                      Microsoft Roslyn C# debugging symbols version 1.0
./libclrjit_unix_x64_x64.so:                  ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.2, BuildID[sha1]=66177aebc4ab51f16fe1e6a5faa90a7ade09b674, stripped
./libclrjit_win_x86_x64.so:                   ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.2, BuildID[sha1]=79ecdf1053497bde0393928dee1a727bc6b6b6a1, stripped
./libclrjit_universal_arm_x64.so:             ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.2, BuildID[sha1]=b15d888e793cca18b6dd42b3b672f7144fbe45ec, stripped
./libjitinterface_x64.so:                     ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.2, BuildID[sha1]=02a2d4a17bcbd0ff35f3caca9252853f95529a3c, stripped
./ILCompiler.TypeSystem.pdb:                  Microsoft Roslyn C# debugging symbols version 1.0
./ILCompiler.DependencyAnalysisFramework.pdb: Microsoft Roslyn C# debugging symbols version 1.0
./ILCompiler.Compiler.pdb:                    Microsoft Roslyn C# debugging symbols version 1.0
./libclrjit_universal_arm64_x64.so:           ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.2, BuildID[sha1]=0b476dc684291af72ab673b77dccbbf7f386cbf8, stripped
./libclrjit_win_x64_x64.so:                   ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.2, BuildID[sha1]=a58827fd1b7ac68612408fa5e13c7db9091938a2, stripped
./ilc.pdb:                                    Microsoft Roslyn C# debugging symbols version 1.0
./ilc:                                        ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=41cb9e347020ad19b6402190528b968b6850c46f, stripped

Regression:

To the best of my knowledge, it has always been this way. This was uncaught until recently when I tried to use a cross built package to bootstrap a native build.

Known Workarounds:

None?

Other info:

TargetOS=linux only appears three places in the .binlog for the build that are after Evaluation. All three are from ILCompiler.cspoj : The first seems to come as a return from ResolveReadyToRunCompilers and the other two (_PrepareForReadyToRunCompilation) and (_CreateR2RImages) use it. NativeAotSupported is reassigned here:

runtime/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj

Line 17 in a5cc707

<NativeAotSupported Condition="'$(TargetOS)' == 'freebsd' and '$(CrossBuild)' == 'true'">false</NativeAotSupported>

but Crossgen2 is still used on the ilc binary and the process does not error from

runtime/src/tasks/Crossgen2Tasks/ResolveReadyToRunCompilers.cs

Lines 116 to 120 in a5cc707

	_targetPlatform != hostPlatform ||
	!GetCrossgenComponentsPaths())
	{
	Log.LogError(Strings.ReadyToRunTargetNotSupportedError);
	return false;

There is no "Property reassignment" note in the binlog when ResolveReadyToRunCompilers changes(?) the TargetOS=freebsd to linux

.binlog is not an allowed attachment type so hopefully a screenshot from the MSBuild Structured Log Viewer is enough to help explain what I am seeing.

The Crossgen2 project seems to avoid the ReadyToRun part during packaging:

runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Crossgen2.sfxproj

Lines 52 to 53 in a5cc707

	<Target Name="RunPublishedCrossgen" AfterTargets="PublishCrossgen"
	Condition="'$(TargetOS)' == '$(HostOS)' and '$(TargetArchitecture)' == '$(BuildArchitecture)' and '$(CrossBuild)' != 'true'">

[dotnet-policy-service]

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

[jkotas]

This was refactored in #103508 . Does the problem still exist in current main?

[Thefrank]

@jkotas

Just checked. It still exists as of e125e93

[jkotas]

Ok, here is the problem:

The regular build uses "last known good" toolchain (the one that comes from https://github.com/dotnet/runtime/blob/main/global.json#L3) for aot compilation of most managed tools and components. The one exception is crossgen2 that uses live built aot toolchain that leads to interesting problems every once in a while.

When cross-compiling from Linux to FreeBSD, there is no "last known good" aot toolchain that is able to cross-compile from Linux to FreeBSD. We end up doing Linux->Linux publish instead due to some incorrect build logic and produce binaries for wrong architecture as you have observed.

I can think about two options for addressing this:

• Enable use of live built aot toolchain for compilation of managed tools and components. The current prior art with crossgen2, is complicated and fragile. We had several attempts to extend it to other tools and components (e.g. Crossgen2 should publish with NativeAOT at the beginning of the build #71557) that we ended up not pursuing. I think that the proper solution would be a multi-stage build: We need to first build the compiler that is then going to be used to compile the shipping compiler. It should be opt-in option and we should keep using "last known good" to keep the default build simple. The source build bootstrapping does similar multi-stage build already, perhaps we can use some of the same controls.
• Disable aot compilation of the managed tools and components when there is no "last known good" toolchain. It may be simpler, but it would mean that the cross-compiling Linux to FreeBSD build does not have full fidelity with the native FreeBSD build.

[Thefrank]

Would disabling AOT for managed items still give FreeBSD (TargetOS) ELF's on components like ILCompiler?
If yes, that might be the simplest solution for the time being.

The multi-stage build sounds like the best solution for this issue. But, unless there is something I am missing, this issue seems limited to a niche case of cross compiling from Linux to FreeBSD/Illumos/Haiku/Other. If it helps other build scenarios for officially supported platforms then there is a better case for this but I am not familiar enough with intricacies here.

[jkotas]

unless there is something I am missing, this issue seems limited to a niche case of cross compiling from Linux to FreeBSD/Illumos/Haiku/Other

I think having the recipe for how to use live-built AOT compilers would be generally useful. We keep having long discussions about whether it is better to use live-built AOT compiler or last-known-good AOT compiler. So having both options and use the one that's more appropriate for given situation would be best.

Would disabling AOT for managed items still give FreeBSD (TargetOS) ELF's on components like ILCompiler?

It would require some extra build logic. Nothing impossible.

[jkotas]

cc @jkoritzinsky @lambdageek