HelloWorld console app build failed on ppc64le with .NET8 Preview3

[Sapana-Khemkar]

Description

HelloWorld console app build failed on ppc64le with .NET8 Preview3 with assert as below

MSBuild version 17.6.0-preview-23174-01+e7de13307 for .NET
* Assertion at /root/sapana/dotnet-s390x/runtime/src/mono/mono/metadata/assembly.c:1084, condition `is_ok (hook_error)' not met, function:mono_assembly_invoke_load_hook_internal, (null) assembly:/home/ubuntu/dotnet-ppc64le-v8.0.0-preview-3/.dotnet/shared/Microsoft.NETCore.App/8.0.0-preview.3.23174.8/System.Private.CoreLib.dll type:NullReferenceException member:(null)

Reproduction Steps

Steps to reproduce issue

1. Cross build .NET8 preview3 on ubuntu18.04 machine and generate tarball for ppc64le
2. Install it on ppc64le vm using below script

mkdir .dotnet
DOTNET_FILE=dotnet-sdk-8.0.100-preview.3.23178.7-linux-ppc64le.tar.gz
export DOTNET_ROOT=$(pwd)/.dotnet
tar zxf "$DOTNET_FILE" -C "$DOTNET_ROOT"
export PATH=$PATH:$DOTNET_ROOT:$DOTNET_ROOT/tools

3. Output of dotnet --info is as below

ubuntu@dotnet2--2:~$ dotnet --info
.NET SDK:
 Version:   8.0.100-preview.3.23178.7
 Commit:    e300b0e1e6

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  20.04
 OS Platform: Linux
 RID:         linux-ppc64le
 Base Path:   /home/ubuntu/dotnet-ppc64le-v8.0.0-preview-3/.dotnet/sdk/8.0.100-preview.3.23178.7/

.NET workloads installed:
There are no installed workloads to display.

Host:
  Version:      8.0.0-preview.3.23174.8
  Architecture: ppc64le
  Commit:       47bad717bd

.NET SDKs installed:
  8.0.100-preview.3.23178.7 [/home/ubuntu/dotnet-ppc64le-v8.0.0-preview-3/.dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 8.0.0-preview.3.23177.8 [/home/ubuntu/dotnet-ppc64le-v8.0.0-preview-3/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 8.0.0-preview.3.23174.8 [/home/ubuntu/dotnet-ppc64le-v8.0.0-preview-3/.dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  DOTNET_ROOT       [/home/ubuntu/dotnet-ppc64le-v8.0.0-preview-3/.dotnet/]

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

4. Create sample console app
   dotnet new console -o HelloWorld
5. Build HelloWorld

cd HelloWorld
dotnet build

Expected behavior

Build HelloWorld console app sucessfully

Actual behavior

Build failed with error

build.log

Regression?

No response

Known Workarounds

None

Configuration

• Which version of .NET is the code running on?

8.0.100-preview.3.23178.7

• What OS and version, and what distro if applicable?

ubuntu@dotnet2--2:~/HelloWorld$ uname -a
Linux dotnet2--2 5.4.0-125-generic #141-Ubuntu SMP Wed Aug 10 13:41:48 UTC 2022 ppc64le ppc64le ppc64le GNU/Linux

Other information

No response

Tagging subscribers to this area:
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

HelloWorld console app build failed on ppc64le with .NET8 Preview3 with assert as below

MSBuild version 17.6.0-preview-23174-01+e7de13307 for .NET
* Assertion at /root/sapana/dotnet-s390x/runtime/src/mono/mono/metadata/assembly.c:1084, condition `is_ok (hook_error)' not met, function:mono_assembly_invoke_load_hook_internal, (null) assembly:/home/ubuntu/dotnet-ppc64le-v8.0.0-preview-3/.dotnet/shared/Microsoft.NETCore.App/8.0.0-preview.3.23174.8/System.Private.CoreLib.dll type:NullReferenceException member:(null)

Reproduction Steps

Steps to reproduce issue

1. Cross build .NET8 preview3 on ubuntu18.04 machine and generate tarball for ppc64le
2. Install it on ppc64le vm using below script

mkdir .dotnet
DOTNET_FILE=dotnet-sdk-8.0.100-preview.3.23178.7-linux-ppc64le.tar.gz
export DOTNET_ROOT=$(pwd)/.dotnet
tar zxf "$DOTNET_FILE" -C "$DOTNET_ROOT"
export PATH=$PATH:$DOTNET_ROOT:$DOTNET_ROOT/tools

3. Output of dotnet --info is as below

ubuntu@dotnet2--2:~$ dotnet --info
.NET SDK:
 Version:   8.0.100-preview.3.23178.7
 Commit:    e300b0e1e6

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  20.04
 OS Platform: Linux
 RID:         linux-ppc64le
 Base Path:   /home/ubuntu/dotnet-ppc64le-v8.0.0-preview-3/.dotnet/sdk/8.0.100-preview.3.23178.7/

.NET workloads installed:
There are no installed workloads to display.

Host:
  Version:      8.0.0-preview.3.23174.8
  Architecture: ppc64le
  Commit:       47bad717bd

.NET SDKs installed:
  8.0.100-preview.3.23178.7 [/home/ubuntu/dotnet-ppc64le-v8.0.0-preview-3/.dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 8.0.0-preview.3.23177.8 [/home/ubuntu/dotnet-ppc64le-v8.0.0-preview-3/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 8.0.0-preview.3.23174.8 [/home/ubuntu/dotnet-ppc64le-v8.0.0-preview-3/.dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  DOTNET_ROOT       [/home/ubuntu/dotnet-ppc64le-v8.0.0-preview-3/.dotnet/]

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

4. Create sample console app
   dotnet new console -o HelloWorld
5. Build HelloWorld

cd HelloWorld
dotnet build

Expected behavior

Build HelloWorld console app sucessfully

Actual behavior

Build failed with error

build.log

Regression?

No response

Known Workarounds

None

Configuration

• Which version of .NET is the code running on?

8.0.100-preview.3.23178.7

• What OS and version, and what distro if applicable?

ubuntu@dotnet2--2:~/HelloWorld$ uname -a
Linux dotnet2--2 5.4.0-125-generic #141-Ubuntu SMP Wed Aug 10 13:41:48 UTC 2022 ppc64le ppc64le ppc64le GNU/Linux

Other information

No response

Author:	Sapana-Khemkar
Assignees:	-
Labels:	untriaged, area-AssemblyLoader-mono, needs-area-label
Milestone:	-

[uweigand]

I'm now seeing this same symptom on s390x as well. However, for some reason it doesn't occur when just doing a "hello world" build, but it requires a more complex msbuild invocation. (E.g. I can always trigger it by just running ./build.sh in the runtime repo.) This seems to be a generic Mono issue.

Some more details I found during debugging:

We are currently attempting to compile some lamba expression in Linq:

          at System.Reflection.Emit.RuntimeAssemblyBuilder:basic_init <0x00064>
          at System.Reflection.Emit.RuntimeAssemblyBuilder:.ctor <0x005a2>
          at System.Reflection.Emit.AssemblyBuilder:DefineDynamicAssembly <0x00256>
          at System.Reflection.Emit.AssemblyBuilder:DefineDynamicAssembly <0x0020e>
          at System.Reflection.Emit.RuntimeAssemblyBuilder:InternalDefineDynamicAssembly <0x00214>
          at System.Reflection.Emit.DynamicMethod:GetDynamicMethodsModule <0x0052e>
          at System.Reflection.Emit.DynamicMethod:Init <0x00820>
          at System.Reflection.Emit.DynamicMethod:.ctor <0x00292>
          at System.Linq.Expressions.Compiler.LambdaCompiler:.ctor <0x00570>
          at System.Linq.Expressions.Compiler.LambdaCompiler:Compile <0x00242>
          at System.Linq.Expressions.LambdaExpression:Compile <0x001e4>

The Mono version of the EmitRuntimeAssemblyBuilder constructor does:

[...]
            basic_init(this);

            // Netcore only allows one module per assembly
            manifest_module = new RuntimeModuleBuilder(this, "RefEmit_InMemoryManifestModule");
            modules = new RuntimeModuleBuilder[] { manifest_module };
[...]

where basic_init is a native icall, where we're running into a failed assertion here:

#12 0x000003ff829979c0 in monoeg_assertion_message (format=<optimized out>) at /home/iii/dotnet-s390x/runtime/src/mono/mono/eglib/goutput.c:207
#13 0x000003ff829c6796 in mono_assembly_invoke_load_hook_internal (alc=0x2aa29d89a60, ass=0x3ff240f1520) at /home/iii/dotnet-s390x/runtime/src/mono/mono/metadata/assembly.c:1084
#14 0x000003ff82a77b46 in mono_reflection_dynimage_basic_init (assemblyb=0x3ff817277a8, error=<optimized out>) at /home/iii/dotnet-s390x/runtime/src/mono/mono/metadata/sre.c:1274
#15 0x000003ff82a7e49e in ves_icall_AssemblyBuilder_basic_init (assemblyb=..., error=0x3ff4a4f6a78) at /home/iii/dotnet-s390x/runtime/src/mono/mono/metadata/sre.c:4474
#16 0x000003ff82a01bc6 in ves_icall_AssemblyBuilder_basic_init_raw (a0=0x3ff4a4f6b68) at /home/iii/dotnet-s390x/runtime/src/mono/mono/metadata/../metadata/icall-def.h:314

because an OnAssemblyLoad callback fails. This callback is Microsoft.Build.BackEnd.Components.RequestBuilder.AssemblyLoadsTracker:CurrentDomainOnAssemblyLoad, which is a relatively recent addition to msbuild, and it does:

        private void CurrentDomainOnAssemblyLoad(object? sender, AssemblyLoadEventArgs args)
        {
            string? assemblyName = args.LoadedAssembly.FullName;
            string assemblyPath = args.LoadedAssembly.IsDynamic ? string.Empty : args.LoadedAssembly.Location;
            Guid mvid = args.LoadedAssembly.ManifestModule.ModuleVersionId;

which throws an exception because ManifestModule returns nil. That exception causes the mono_assembly_invoke_load_hook_internal routine to fail an assertion.

Now, looking back up at the EmitRuntimeAssemblyBuilder constructor we see that the manifest_module variable (which is what gets returned as ManifestModule) is only initialized after the basic_init call, but the OnAssemblyLoad hook, which is called from within basic_init, already attempts to read it. This seems like some sort of ordering issue.

Not sure if there is an inconsistency between the way Mono and CoreCLR handle these initialization steps here. CC @jkotas @vargaz @directhex - any comments?

[jkotas]

Yes, it looks like an ordering issue on Mono. CoreCLR gives fully initialized instance to the callback.

[vargaz]

Why does this only fail on s390x ?

[jkotas]

No other platform runs .NET SDK on Mono.

[vargaz]

Is it possible to reproduce this on non-s390x ? I tried setting PrimaryRuntimeFlavor to mono but it doesn't seem to be enough. What is the build.sh invocation which fails ?