#65738 causes SIGSEGV running helloworld on linux-x86 preview 3

[ta264]

Description

I can build a functional SDK for linux-x86 for .NET 7 Preview 2 once I fix #68044 and disable ReadyToRun (which causes other issues I will dig into next)

This can run a hello world application OK.

After updating to preview 3, it fails with SIGSEGV. I bisected and the failure is introduced by eb8460f from #65738

If I revert this commit and rebuild preview 3 then everything works again - see the pipeline here:
https://github.com/Servarr/dotnet-linux-x86/tree/4b0474c30e5ea2900e14fbe94831d64d7f44b318

Reproduction Steps

On a linux-x64 host with a preview 3 SDK, run dotnet new console && dotnet build

Build .NET 7 preview 3 runtime for linux-x86. Example pipeline is here:
https://github.com/Servarr/dotnet-linux-x86/tree/4b0474c30e5ea2900e14fbe94831d64d7f44b318

Remove this line reverting the commit causing the issue:
https://github.com/Servarr/dotnet-linux-x86/blob/4b0474c30e5ea2900e14fbe94831d64d7f44b318/azure-pipelines.yml#L77

Run the output on the linux-x86 host (I'm using a ubuntu 20.04 docker with mulitlib support enabled) using the runtime generated above.

Expected behavior

Hello, World!

Actual behavior

(gdb) run ./bin/Debug/net7.0/helloworld2.dll
Starting program: /git_working/dotnet/dotnet ./bin/Debug/net7.0/helloworld2.dll
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0xf735db40 (LWP 300)]
[New Thread 0xf69ffb40 (LWP 301)]
[New Thread 0xf5fffb40 (LWP 302)]
[New Thread 0xf55ffb40 (LWP 303)]
[New Thread 0xf1bfeb40 (LWP 304)]

Thread 1 "dotnet" received signal SIGSEGV, Segmentation fault.
0xf7474e7a in FixupPrecode::GenerateCodePage (pageBase=0xf6186000 "\377%", pageBaseRX=0xf6186000 "\377%") at /runtime/src/coreclr/vm/precode.cpp:738
738             *(BYTE**)(pageBase + i + SYMBOL_VALUE(FixupPrecodeCode_Target_Offset)) = pTargetSlot;
(gdb) bt
#0  0xf7474e7a in FixupPrecode::GenerateCodePage (pageBase=0xf6186000 "\377%", pageBaseRX=0xf6186000 "\377%") at /runtime/src/coreclr/vm/precode.cpp:738
#1  0xf76b6cc7 in UnlockedLoaderHeap::UnlockedReservePages (this=0xf79eb234 <g_pSystemDomainMemory+1956>, dwSizeToCommit=8192) at /runtime/src/coreclr/utilcode/loaderheap.cpp:1192
#2  0xf76b6efd in UnlockedLoaderHeap::GetMoreCommittedPages (this=0xf79eb234 <g_pSystemDomainMemory+1956>, dwMinSize=<optimized out>) at /runtime/src/coreclr/utilcode/loaderheap.cpp:1362
#3  0xf76b741f in UnlockedLoaderHeap::UnlockedAllocAlignedMem_NoThrow (this=0xf79eb234 <g_pSystemDomainMemory+1956>, dwRequestedSize=216, alignment=1, pdwExtra=0xffffa794) at /runtime/src/coreclr/utilcode/loaderheap.cpp:1751
#4  UnlockedLoaderHeap::UnlockedAllocAlignedMem (this=0xf79eb234 <g_pSystemDomainMemory+1956>, dwRequestedSize=216, dwAlignment=1, pdwExtra=0xffffa794) at /runtime/src/coreclr/utilcode/loaderheap.cpp:1856
#5  0xf73b68c6 in LoaderHeap::RealAllocAlignedMem (this=0xf79eb230 <g_pSystemDomainMemory+1952>, dwRequestedSize=216, dwAlignment=1) at /runtime/src/coreclr/inc/loaderheap.h:664
#6  0xf7474b51 in Precode::AllocateTemporaryEntryPoints (pChunk=0xf6b24d1c, pLoaderAllocator=0xf79eb0cc <g_pSystemDomainMemory+1596>, pamTracker=0xffffcc18) at /runtime/src/coreclr/vm/precode.cpp:484
#7  0xf744cc18 in MethodDescChunk::CreateTemporaryEntryPoints (this=0xf6b24d1c, pLoaderAllocator=0xf79eb0cc <g_pSystemDomainMemory+1596>, pamTracker=0xffffcc18) at /runtime/src/coreclr/vm/method.cpp:2964
#8  0xf7571623 in MethodDescChunk::EnsureTemporaryEntryPointsCreated (this=0xf6b24d1c, pLoaderAllocator=0xf6187000, pamTracker=0xff0) at /runtime/src/coreclr/vm/method.hpp:2211
#9  MethodTableBuilder::SetupMethodTable2 (this=0xffffcaa8, pLoaderModule=0xf6b20000) at /runtime/src/coreclr/vm/methodtablebuilder.cpp:10645
#10 0xf75684a1 in MethodTableBuilder::BuildMethodTableThrowing (this=0xffffcaa8, pAllocator=0xf79eb0cc <g_pSystemDomainMemory+1596>, pLoaderModule=0xf6b20000, pModule=0xf6b20000, cl=33554530, pBuildingInterfaceList=0x0, pLayoutRawFieldInfos=0x0, pParentMethodTable=0x0, bmtGenericsInfo=0xffffca60, parentInst=..., cBuildingInterfaceList=0) at /runtime/src/coreclr/vm/methodtablebuilder.cpp:1774
#11 0xf7579b01 in ClassLoader::CreateTypeHandleForTypeDefThrowing (pModule=0xf6b20000, cl=33554530, inst=..., pamTracker=0xffffcc18) at /runtime/src/coreclr/vm/methodtablebuilder.cpp:12391
#12 0xf73d7d39 in ClassLoader::CreateTypeHandleForTypeKey (pKey=0xffffce80, pamTracker=0xffffcc18) at /runtime/src/coreclr/vm/clsload.cpp:2915
#13 0xf73d77a0 in ClassLoader::DoIncrementalLoad (pTypeKey=0xffffce80, typeHnd=..., currentLevel=CLASS_LOAD_BEGIN) at /runtime/src/coreclr/vm/clsload.cpp:2846
#14 0xf73d875a in ClassLoader::LoadTypeHandleForTypeKey_Body (this=0x565e8ea0, pTypeKey=0xffffce80, typeHnd=..., targetLevel=CLASS_LOAD_EXACTPARENTS) at /runtime/src/coreclr/vm/clsload.cpp:3531
#15 0xf73d4bb4 in ClassLoader::LoadTypeHandleForTypeKey (this=0x565e8ea0, pTypeKey=0xffffce80, typeHnd=..., targetLevel=CLASS_LOADED, pInstContext=0x0) at /runtime/src/coreclr/vm/clsload.cpp:3250
#16 0xf73d5c7a in ClassLoader::LoadTypeDefThrowing (pModule=0xf6b20000, typeDef=33554530, fNotFoundAction=ClassLoader::ReturnNullIfNotFound, fUninstantiated=ClassLoader::PermitUninstDefOrRef, tokenNotToLoad=0, level=CLASS_LOADED, pTargetInstantiation=0x0) at /runtime/src/coreclr/vm/clsload.cpp:2222
#17 0xf73d28ab in ClassLoader::LoadTypeHandleThrowing (this=0x565e8ea0, pName=0xffffcf60, level=CLASS_LOADED, pLookInThisModuleOnly=0x0) at /runtime/src/coreclr/vm/clsload.cpp:1469
#18 0xf73d2591 in ClassLoader::LoadTypeByNameThrowing (pAssembly=0x565e8e60, nameSpace=0xf7883ca5 "System", name=0xf789d947 "Object", fNotFound=ClassLoader::ThrowIfNotFound, fLoadTypes=ClassLoader::LoadTypes, level=CLASS_LOADED) at /runtime/src/coreclr/vm/assembly.hpp:113
#19 0xf73b07c8 in CoreLibBinder::LookupClassLocal (this=<optimized out>, id=CLASS__ELEMENT_TYPE_OBJECT) at /runtime/src/coreclr/vm/binder.cpp:65
#20 CoreLibBinder::LookupClass (id=CLASS__ELEMENT_TYPE_OBJECT) at /runtime/src/coreclr/vm/binder.cpp:31
#21 0xf739a58d in CoreLibBinder::GetClass (id=CLASS__ELEMENT_TYPE_OBJECT) at /runtime/src/coreclr/vm/binder.h:341
#22 SystemDomain::LoadBaseSystemClasses (this=0xf79eaa90 <g_pSystemDomainMemory>) at /runtime/src/coreclr/vm/appdomain.cpp:1329
#23 0xf739a315 in SystemDomain::Init (this=0xf79eaa90 <g_pSystemDomainMemory>) at /runtime/src/coreclr/vm/appdomain.cpp:1198
#24 0xf77ea59c in EEStartupHelper () at /runtime/src/coreclr/vm/ceemain.cpp:965
#25 0xf77e9870 in EEStartup()::$_0::operator()(void*) const (this=<optimized out>, p=<optimized out>) at /runtime/src/coreclr/vm/ceemain.cpp:1111
#26 EEStartup () at /runtime/src/coreclr/vm/ceemain.cpp:1113
#27 0xf77e9781 in EnsureEEStarted () at /runtime/src/coreclr/vm/ceemain.cpp:314
#28 0xf73e1ca2 in CorHost2::Start (this=0x56578920) at /runtime/src/coreclr/vm/corhost.cpp:101
#29 0xf73972f9 in coreclr_initialize (exePath=0x56570190 "/git_working/dotnet/dotnet", appDomainFriendlyName=0xf7a47ae4 "clrhost", propertyCount=9, propertyKeys=0x56581420, propertyValues=0x56581b40, hostHandle=0xffffd264, domainId=0xffffd260) at /runtime/src/coreclr/dlls/mscoree/exports.cpp:251
#30 0xf7a1346f in ?? () from /git_working/dotnet/shared/Microsoft.NETCore.App/7.0.0-preview.3.22175.4/libhostpolicy.so
#31 0xf7a21209 in ?? () from /git_working/dotnet/shared/Microsoft.NETCore.App/7.0.0-preview.3.22175.4/libhostpolicy.so
#32 0xf7a20aed in corehost_main () from /git_working/dotnet/shared/Microsoft.NETCore.App/7.0.0-preview.3.22175.4/libhostpolicy.so
#33 0xf7a66f62 in ?? () from /git_working/dotnet/host/fxr/7.0.0-preview.3.22175.4/libhostfxr.so
#34 0xf7a65a4f in ?? () from /git_working/dotnet/host/fxr/7.0.0-preview.3.22175.4/libhostfxr.so
#35 0xf7a624f6 in hostfxr_main_startupinfo () from /git_working/dotnet/host/fxr/7.0.0-preview.3.22175.4/libhostfxr.so
#36 0x56562ca2 in ?? ()
#37 0x56562f18 in ?? ()
#38 0xf7ac7ee5 in __libc_start_main () from /lib/i386-linux-gnu/libc.so.6
#39 0x565588df in ?? ()

Regression?

This is a regression from .NET 7 Preview 2

Known Workarounds

Rebuild Preview 3 with eb8460fd29f reverted

Configuration

.NET 7 Preview 3
Cross compiling on Ubuntu 20.04 for linux-x86
Running the output in an Ubuntu 20.04 docker with multilib support enabled

dpkg --add-architecture i386
apt-get update
apt-get install libc6:i386 libgcc1:i386 libgssapi-krb5-2:i386 libicu66:i386 libssl1.1:i386 libstdc++6:i386 zlib1g:i386

Other information

I think the issue is introduced by eb8460f from #65738 authored by @janvorli (I hope this is not inappropriate, apologies in advance if I shouldn't have mentioned you)

[ta264]

I can also confirm the same issue seems to persist in main as of 0e18cfd

Stack trace is:

(gdb) run bin/Debug/net7.0/helloworld2.dll
Starting program: /git_working/dotnet/dotnet bin/Debug/net7.0/helloworld2.dll
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0xf7365b40 (LWP 318)]
[New Thread 0xf69ffb40 (LWP 319)]
[New Thread 0xf5fffb40 (LWP 320)]
[New Thread 0xf55ffb40 (LWP 321)]
[New Thread 0xf1bfeb40 (LWP 322)]

Thread 1 "dotnet" received signal SIGSEGV, Segmentation fault.
0xf747b8aa in FixupPrecode::GenerateCodePage (pageBase=0xf618e000 "\377%", pageBaseRX=0xf618e000 "\377%") at /runtime/src/coreclr/vm/precode.cpp:747
747             *(BYTE**)(pageBase + i + SYMBOL_VALUE(FixupPrecodeCode_Target_Offset)) = pTargetSlot;
(gdb) bt
#0  0xf747b8aa in FixupPrecode::GenerateCodePage (pageBase=0xf618e000 "\377%", pageBaseRX=0xf618e000 "\377%") at /runtime/src/coreclr/vm/precode.cpp:747
#1  0xf76bade7 in UnlockedLoaderHeap::UnlockedReservePages (this=0xf79eb234 <g_pSystemDomainMemory+1956>, dwSizeToCommit=8192) at /runtime/src/coreclr/utilcode/loaderheap.cpp:1192
#2  0xf76bb01d in UnlockedLoaderHeap::GetMoreCommittedPages (this=0xf79eb234 <g_pSystemDomainMemory+1956>, dwMinSize=<optimized out>) at /runtime/src/coreclr/utilcode/loaderheap.cpp:1362
#3  0xf76bb53f in UnlockedLoaderHeap::UnlockedAllocAlignedMem_NoThrow (this=0xf79eb234 <g_pSystemDomainMemory+1956>, dwRequestedSize=216, alignment=1, pdwExtra=0xffffa7b4) at /runtime/src/coreclr/utilcode/loaderheap.cpp:1751
#4  UnlockedLoaderHeap::UnlockedAllocAlignedMem (this=0xf79eb234 <g_pSystemDomainMemory+1956>, dwRequestedSize=216, dwAlignment=1, pdwExtra=0xffffa7b4) at /runtime/src/coreclr/utilcode/loaderheap.cpp:1856
#5  0xf73be6d6 in LoaderHeap::RealAllocAlignedMem (this=0xf79eb230 <g_pSystemDomainMemory+1952>, dwRequestedSize=216, dwAlignment=1) at /runtime/src/coreclr/inc/loaderheap.h:664
#6  0xf747b581 in Precode::AllocateTemporaryEntryPoints (pChunk=0xf6b2cdb4, pLoaderAllocator=0xf79eb0cc <g_pSystemDomainMemory+1596>, pamTracker=0xffffcc38) at /runtime/src/coreclr/vm/precode.cpp:484
#7  0xf7453b38 in MethodDescChunk::CreateTemporaryEntryPoints (this=0xf6b2cdb4, pLoaderAllocator=0xf79eb0cc <g_pSystemDomainMemory+1596>, pamTracker=0xffffcc38) at /runtime/src/coreclr/vm/method.cpp:2964
#8  0xf7576733 in MethodDescChunk::EnsureTemporaryEntryPointsCreated (this=0xf6b2cdb4, pLoaderAllocator=0xf618f000, pamTracker=0xff0) at /runtime/src/coreclr/vm/method.hpp:2211
#9  MethodTableBuilder::SetupMethodTable2 (this=0xffffcac8, pLoaderModule=0xf6b28000) at /runtime/src/coreclr/vm/methodtablebuilder.cpp:10645
#10 0xf756d5b1 in MethodTableBuilder::BuildMethodTableThrowing (this=0xffffcac8, pAllocator=0xf79eb0cc <g_pSystemDomainMemory+1596>, pLoaderModule=0xf6b28000, pModule=0xf6b28000, cl=33554530, pBuildingInterfaceList=0x0, pLayoutRawFieldInfos=0x0, pParentMethodTable=0x0, bmtGenericsInfo=0xffffca80, parentInst=..., cBuildingInterfaceList=0) at /runtime/src/coreclr/vm/methodtablebuilder.cpp:1774
#11 0xf757ec11 in ClassLoader::CreateTypeHandleForTypeDefThrowing (pModule=0xf6b28000, cl=33554530, inst=..., pamTracker=0xffffcc38) at /runtime/src/coreclr/vm/methodtablebuilder.cpp:12391
#12 0xf73dfa39 in ClassLoader::CreateTypeHandleForTypeKey (pKey=0xffffcea0, pamTracker=0xffffcc38) at /runtime/src/coreclr/vm/clsload.cpp:2915
#13 0xf73df4a0 in ClassLoader::DoIncrementalLoad (pTypeKey=0xffffcea0, typeHnd=..., currentLevel=CLASS_LOAD_BEGIN) at /runtime/src/coreclr/vm/clsload.cpp:2846
#14 0xf73e045a in ClassLoader::LoadTypeHandleForTypeKey_Body (this=0x565e9fc0, pTypeKey=0xffffcea0, typeHnd=..., targetLevel=CLASS_LOAD_EXACTPARENTS) at /runtime/src/coreclr/vm/clsload.cpp:3531
#15 0xf73dc8b4 in ClassLoader::LoadTypeHandleForTypeKey (this=0x565e9fc0, pTypeKey=0xffffcea0, typeHnd=..., targetLevel=CLASS_LOADED, pInstContext=0x0) at /runtime/src/coreclr/vm/clsload.cpp:3250
#16 0xf73dd97a in ClassLoader::LoadTypeDefThrowing (pModule=0xf6b28000, typeDef=33554530, fNotFoundAction=ClassLoader::ReturnNullIfNotFound, fUninstantiated=ClassLoader::PermitUninstDefOrRef, tokenNotToLoad=0, level=CLASS_LOADED, pTargetInstantiation=0x0) at /runtime/src/coreclr/vm/clsload.cpp:2222
#17 0xf73da6db in ClassLoader::LoadTypeHandleThrowing (this=0x565e9fc0, pName=0xffffcf80, level=CLASS_LOADED, pLookInThisModuleOnly=0x0) at /runtime/src/coreclr/vm/clsload.cpp:1469
#18 0xf73da3c1 in ClassLoader::LoadTypeByNameThrowing (pAssembly=0x565e9f80, nameSpace=0xf7885ca5 "System", name=0xf789faa7 "Object", fNotFound=ClassLoader::ThrowIfNotFound, fLoadTypes=ClassLoader::LoadTypes, level=CLASS_LOADED) at /runtime/src/coreclr/vm/assembly.hpp:113
#19 0xf73b8818 in CoreLibBinder::LookupClassLocal (this=<optimized out>, id=CLASS__ELEMENT_TYPE_OBJECT) at /runtime/src/coreclr/vm/binder.cpp:65
#20 CoreLibBinder::LookupClass (id=CLASS__ELEMENT_TYPE_OBJECT) at /runtime/src/coreclr/vm/binder.cpp:31
#21 0xf73a257d in CoreLibBinder::GetClass (id=CLASS__ELEMENT_TYPE_OBJECT) at /runtime/src/coreclr/vm/binder.h:341
#22 SystemDomain::LoadBaseSystemClasses (this=0xf79eaa90 <g_pSystemDomainMemory>) at /runtime/src/coreclr/vm/appdomain.cpp:1329
#23 0xf73a2305 in SystemDomain::Init (this=0xf79eaa90 <g_pSystemDomainMemory>) at /runtime/src/coreclr/vm/appdomain.cpp:1198
#24 0xf77ed7d2 in EEStartupHelper () at /runtime/src/coreclr/vm/ceemain.cpp:966
#25 0xf77eca90 in EEStartup()::$_0::operator()(void*) const (this=<optimized out>, p=<optimized out>) at /runtime/src/coreclr/vm/ceemain.cpp:1112
#26 EEStartup () at /runtime/src/coreclr/vm/ceemain.cpp:1114
#27 0xf77ec9a1 in EnsureEEStarted () at /runtime/src/coreclr/vm/ceemain.cpp:314
#28 0xf73e9822 in CorHost2::Start (this=0x565792d0) at /runtime/src/coreclr/vm/corhost.cpp:101
#29 0xf739f2e9 in coreclr_initialize (exePath=0x56571190 "/git_working/dotnet/dotnet", appDomainFriendlyName=0xf7a47ae4 "clrhost", propertyCount=9, propertyKeys=0x56582630, propertyValues=0x56582dc0, hostHandle=0xffffd264, domainId=0xffffd260) at /runtime/src/coreclr/dlls/mscoree/exports.cpp:251
#30 0xf7a1346f in ?? () from /git_working/dotnet/shared/Microsoft.NETCore.App/7.0.0-preview.5.22222.99/libhostpolicy.so
#31 0xf7a21209 in ?? () from /git_working/dotnet/shared/Microsoft.NETCore.App/7.0.0-preview.5.22222.99/libhostpolicy.so
#32 0xf7a20aed in corehost_main () from /git_working/dotnet/shared/Microsoft.NETCore.App/7.0.0-preview.5.22222.99/libhostpolicy.so
#33 0xf7a66f82 in ?? () from /git_working/dotnet/host/fxr/7.0.0-preview.5.22222.99/libhostfxr.so
#34 0xf7a65a9f in ?? () from /git_working/dotnet/host/fxr/7.0.0-preview.5.22222.99/libhostfxr.so
#35 0xf7a62536 in hostfxr_main_startupinfo () from /git_working/dotnet/host/fxr/7.0.0-preview.5.22222.99/libhostfxr.so
#36 0x56562d62 in ?? ()
#37 0x56562fd8 in ?? ()
#38 0xf7ac7ee5 in __libc_start_main () from /lib/i386-linux-gnu/libc.so.6
#39 0x565588df in ?? ()

[janvorli]

@ta264 thank you for reporting this issue, you are definitely right and the issue is caused by my changes. I'll fix it.

[ta264]

Thanks!

[janvorli]

@ta264 it is strange, but it works ok for me (tested with build from the main branch). I have set a breakpoint at the FixupPrecode::GenerateCodePage and it has passed multiple times and generated expected code. I wonder, which clang version have you used to build it?

[janvorli]

I was using clang 9, which is the one we use for official builds.

[ta264]

@janvorli thanks very much for investigating

I was using clang 9. This is the docker file I was using to generate the build environment and crossrootfs:
https://github.com/Servarr/dotnet-linux-x86/blob/4b0474c30e5ea2900e14fbe94831d64d7f44b318/Dockerfile#L8

Then I make the patches listed here:
https://github.com/Servarr/dotnet-linux-x86/blob/4b0474c30e5ea2900e14fbe94831d64d7f44b318/azure-pipelines.yml#L75-L83

Then build runtime in two stages:
https://github.com/Servarr/dotnet-linux-x86/blob/4b0474c30e5ea2900e14fbe94831d64d7f44b318/azure-pipelines.yml#L93-L94

Which mirrors how we had to do it on freebsd to get everything working.

Finally, I'm testing with dotnet helloworld.dll (rather than corerun)

[ta264]

Can I ask how you're building it? It's very possible my edits to make it build are having unintended consequences

[janvorli]

I've used the sed and options to the build commands from your https://github.com/Servarr/dotnet-linux-x86/blob/4b0474c30e5ea2900e14fbe94831d64d7f44b318/azure-pipelines.yml from the Runtime part. I've just executed them manually and without a docker and without the official build id passed in. And using the build.sh from the root of the repo, not from the eng folder.
I've built the rootfs for x86 crossbuild using the eng/common/cross/build-rootfs.sh, but that was one I've built a long time ago. It was a rootfs for Ubuntu 18.04. Built like this, if I remember it correctly:

ROOTFS_DIR=~/rootfs/x86 ./eng/common/cross/build-rootfs.sh x86 bionic

I was running it in a docker container created from i386/ubuntu:18.04.

Would you be able to share a disassembly of the code of the crashing method? Just running the GDB disass command at the crash. And also the values of the registers dumped using info registers command would be helpful.

[ta264]

disass output:

Dump of assembler code for function FixupPrecode::GenerateCodePage(unsigned char*, unsigned char*):                                                                                                                                             [0/1333]
   0xf747b810 <+0>:     push   %ebp
   0xf747b811 <+1>:     mov    %esp,%ebp
   0xf747b813 <+3>:     push   %ebx
   0xf747b814 <+4>:     push   %edi
   0xf747b815 <+5>:     push   %esi
   0xf747b816 <+6>:     sub    $0xc,%esp
   0xf747b819 <+9>:     call   0xf747b81e <FixupPrecode::GenerateCodePage(unsigned char*, unsigned char*)+14>
   0xf747b81e <+14>:    pop    %ebx
   0xf747b81f <+15>:    add    $0x55c40a,%ebx
   0xf747b825 <+21>:    call   0xf76d0210 <GetOsPageSize()>
   0xf747b82a <+26>:    mov    %eax,%ecx
   0xf747b82c <+28>:    mov    $0x2aaaaaab,%edx
   0xf747b831 <+33>:    imul   %edx
   0xf747b833 <+35>:    mov    %edx,%eax
   0xf747b835 <+37>:    shr    $0x1f,%eax
   0xf747b838 <+40>:    shr    $0x2,%edx
   0xf747b83b <+43>:    add    %eax,%edx
   0xf747b83d <+45>:    shl    $0x3,%edx
   0xf747b840 <+48>:    lea    (%edx,%edx,2),%edi
   0xf747b843 <+51>:    test   %edi,%edi
   0xf747b845 <+53>:    jle    0xf747b8ca <FixupPrecode::GenerateCodePage(unsigned char*, unsigned char*)+186>
   0xf747b84b <+59>:    mov    0x8(%ebp),%edx
   0xf747b84e <+62>:    add    0xc(%ebp),%ecx
   0xf747b851 <+65>:    mov    0x570(%ebx),%eax
   0xf747b857 <+71>:    add    %edx,%eax
   0xf747b859 <+73>:    mov    %eax,-0x18(%ebp)
   0xf747b85c <+76>:    mov    0x116c(%ebx),%eax
   0xf747b862 <+82>:    add    %edx,%eax
   0xf747b864 <+84>:    mov    %eax,-0x14(%ebp)
   0xf747b867 <+87>:    mov    0x11f8(%ebx),%eax
   0xf747b86d <+93>:    add    %edx,%eax
   0xf747b86f <+95>:    mov    %eax,-0x10(%ebp)
   0xf747b872 <+98>:    mov    0x1038(%ebx),%ebx
   0xf747b878 <+104>:   xor    %esi,%esi
   0xf747b87a <+106>:   nop
   0xf747b87b <+107>:   nop
   0xf747b87c <+108>:   nop
   0xf747b87d <+109>:   nop
   0xf747b87e <+110>:   nop
   0xf747b87f <+111>:   nop
   0xf747b880 <+112>:   movsd  0x10(%ebx),%xmm0
   0xf747b885 <+117>:   mov    0x8(%ebp),%eax
   0xf747b888 <+120>:   movsd  %xmm0,0x10(%eax,%esi,1)
   0xf747b88e <+126>:   movsd  (%ebx),%xmm0
   0xf747b892 <+130>:   movsd  0x8(%ebx),%xmm1
   0xf747b897 <+135>:   movsd  %xmm1,0x8(%eax,%esi,1)
   0xf747b89d <+141>:   movsd  %xmm0,(%eax,%esi,1)
   0xf747b8a2 <+146>:   mov    %edi,%eax
   0xf747b8a4 <+148>:   lea    (%ecx,%esi,1),%edi
   0xf747b8a7 <+151>:   mov    -0x10(%ebp),%edx
=> 0xf747b8aa <+154>:   mov    %edi,(%edx,%esi,1)
   0xf747b8ad <+157>:   lea    0x4(%ecx,%esi,1),%edi
   0xf747b8b1 <+161>:   mov    -0x14(%ebp),%edx
   0xf747b8b4 <+164>:   mov    %edi,(%edx,%esi,1)
   0xf747b8b7 <+167>:   lea    0x8(%ecx,%esi,1),%edi
   0xf747b8bb <+171>:   mov    -0x18(%ebp),%edx
   0xf747b8be <+174>:   mov    %edi,(%edx,%esi,1)
   0xf747b8c1 <+177>:   mov    %eax,%edi
   0xf747b8c3 <+179>:   add    $0x18,%esi
   0xf747b8c6 <+182>:   cmp    %eax,%esi
   0xf747b8c8 <+184>:   jl     0xf747b880 <FixupPrecode::GenerateCodePage(unsigned char*, unsigned char*)+112>
   0xf747b8ca <+186>:   add    $0xc,%esp
   0xf747b8cd <+189>:   pop    %esi
   0xf747b8ce <+190>:   pop    %edi
   0xf747b8cf <+191>:   pop    %ebx
   0xf747b8d0 <+192>:   pop    %ebp
   0xf747b8d1 <+193>:   ret

info registers output:

eax            0xff0               4080
ecx            0xf617f000          -166203392
edx            0xed4f0002          -313589758
ebx            0xf76a8ea9          -144011607
esp            0xffffa6c0          0xffffa6c0
ebp            0xffffa6d8          0xffffa6d8
esi            0x0                 0
edi            0xf617f000          -166203392
eip            0xf747b8aa          0xf747b8aa <FixupPrecode::GenerateCodePage(unsigned char*, unsigned char*)+154>
eflags         0x10246             [ PF ZF IF RF ]
cs             0x23                35
ss             0x2b                43
ds             0x2b                43
es             0x2b                43
fs             0x0                 0
gs             0x63                99

Let me know if I can provide anything else, happy to help.

I tried running under i386/ubuntu:18.04 but had the same issue. The other difference you mention is using an 18.04 rootfs, I think mine is 16.04. I'll try with an 18.04 one.

[ta264]

I'm struggling to get it to build under an 18.04 rootfs, debootstrap is failing in docker and native I end up with

[ 82%] Building C object mono/mini/CMakeFiles/monosgen-objects.dir/home/ta264/git_working/runtime/src/native/libs/System.Globalization.Native/pal_icushim.c.o                                              [ 82%] Building C object mono/mini/CMakeFiles/monosgen-objects.dir/main-core.c.o
  In file included from /home/ta264/git_working/runtime/src/native/libs/System.Globalization.Native/pal_icushim.c:16:
  /home/ta264/git_working/rootfs/x86/usr/include/stdio.h:41:10: fatal error: 'bits/libio.h' file not found                                #include <bits/libio.h>

Any tips appreciated!

[janvorli]

Here are the exact steps I've used to build the rootfs and then build the runtime using it from the main branch. I have tried it from scratch to make sure nothing stale is left in my repo by accident. I am running the build on x64 Ubuntu 18.04 native box. First checkout the dotnet/runtime repo and apply the "sed" changes that you've shared:
https://github.com/Servarr/dotnet-linux-x86/blob/4b0474c30e5ea2900e14fbe94831d64d7f44b318/azure-pipelines.yml#L79-L83

In the dotnet/arcade repo:

sudo ROOTFS_DIR=/home/janvorli/rootfs/x86.latest eng/common/cross/build-rootfs.sh x86 bionic

In the dotnet/runtime repo:

ROOTFS_DIR=/home/janvorli/rootfs/x86.latest ./build.sh -c Release -cross -os Linux -arch x86 -clang9 -subset Clr.Native+Host.Native
ROOTFS_DIR=/home/janvorli/rootfs/x86.latest ./build.sh -c Release -cross -os Linux -arch x86 -clang9 /p:AppHostSourcePath=/home/janvorli/git/runtime/artifacts/obj/linux-x86.Release/apphost/standalone/apphost

This completes cleanly for me. Since all the platform specific libraries should come from the rootfs, I would expect it to work for you the same way.

[ta264]

Thanks a lot for the details. I'll investigate further and let you know.