Statically Linked NativeAOT HelloWorld throws Segmentation fault with Globalization Cultural Data

[Beau-Gosse-dev]

Description

Trying to test statically linking a basic hello world app on Ubuntu 20.04 and found that it throws a seg fault inside the globalization code.

Reproduction Steps

(On Ubuntu 20.04.4 LTS x64)
dotnet new console
dotnet add package Microsoft.Dotnet.ILCompiler --prerelease
dotnet publish -r linux-x64 -c Release /p:StaticallyLinked=true --self-contained
./bin/Release/net7.0/linux-x64/native/HelloWorld

Expected behavior

Output "Hello, World!"

Actual behavior

Thread 1 "HelloWorld" received signal SIGSEGV, Segmentation fault.
0x00007fffd74b6603 in void std::call_once<void (&)()>(std::once_flag&, void (&)()) () from /lib/x86_64-linux-gnu/libicuuc.so.66
(gdb) bt
#0  0x00007fffd74b6603 in void std::call_once<void (&)()>(std::once_flag&, void (&)()) () from /lib/x86_64-linux-gnu/libicuuc.so.66
#1  0x00007fffd74b63b4 in icu_66::umtx_initImplPreInit(icu_66::UInitOnce&) () from /lib/x86_64-linux-gnu/libicuuc.so.66
#2  0x00007fffd74e79cb in ?? () from /lib/x86_64-linux-gnu/libicuuc.so.66
#3  0x00007fffd52c0a60 in ulocdata_getCLDRVersion_66 () from /lib/x86_64-linux-gnu/libicui18n.so.66
#4  0x0000000000476de2 in ValidateICUDataCanLoad () at /__w/1/s/src/native/libs/System.Globalization.Native/pal_icushim.c:375
#5  GlobalizationNative_LoadICU () at /__w/1/s/src/native/libs/System.Globalization.Native/pal_icushim.c:471
#6  0x0000000000633c5c in S_P_CoreLib_System_Globalization_GlobalizationMode_Settings___cctor ()
#7  0x00000000006209a9 in S_P_CoreLib_System_Runtime_CompilerServices_ClassConstructorRunner__EnsureClassConstructorRun ()
#8  0x00000000006208b9 in S_P_CoreLib_System_Runtime_CompilerServices_ClassConstructorRunner__CheckStaticClassConstructionReturnNonGCStaticBase ()
#9  0x00000000005cd103 in S_P_CoreLib_System_Globalization_CultureData__CreateCultureWithInvariantData ()
#10 0x00000000005cd1ca in S_P_CoreLib_System_Globalization_CultureData__get_Invariant ()
#11 0x00000000005ddf7c in S_P_CoreLib_System_Globalization_TextInfo___cctor ()
#12 0x00000000006209a9 in S_P_CoreLib_System_Runtime_CompilerServices_ClassConstructorRunner__EnsureClassConstructorRun ()
#13 0x00000000006208a9 in S_P_CoreLib_System_Runtime_CompilerServices_ClassConstructorRunner__CheckStaticClassConstructionReturnGCStaticBase ()
#14 0x000000000064cd17 in System_Console_System_Text_EncodingHelper__GetCharset ()
#15 0x000000000064cc29 in System_Console_System_Text_EncodingHelper__GetEncodingFromCharset ()
#16 0x0000000000645c09 in System_Console_System_ConsolePal__GetConsoleEncoding ()
#17 0x0000000000644cf4 in System_Console_System_Console__get_OutputEncoding ()
#18 0x0000000000644d9b in System_Console_System_Console__CreateOutputWriter ()
#19 0x0000000000644f43 in System_Console_System_Console___get_Out_g__EnsureInitialized_26_0 ()
#20 0x0000000000644e80 in System_Console_System_Console__WriteLine_12 ()
#21 0x00000000006e9f1f in __managed__Main ()
#22 0x00000000004058d9 in main (argc=1, argv=0x7fffffffe478) at /__w/1/s/src/coreclr/nativeaot/Bootstrap/main.cpp:205

Regression?

No response

Known Workarounds

Either disable globalization with <InvariantGlobalization>true</InvariantGlobalization> or don't statically link (remove /p:StaticallyLinked=true argument)

Configuration

dotnet version: 7.0.100-preview.3.22179.4

OS: Ubuntu 20.04.4 LTS

Architecture: x64

Other information

Reproduces on Ubuntu, but not AmazonLinux2. Ubuntu has libicuuc.so.66 but AmazonLinux2 has libicuuc.so.50

We can take a look at this, but were wondering if it was already known.

Tagging subscribers to this area: @dotnet/area-system-globalization
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Trying to test statically linking a basic hello world app on Ubuntu 20.04 and found that it throws a seg fault inside the globalization code.

Reproduction Steps

(On Ubuntu 20.04.4 LTS)
dotnet new console
dotnet add package Microsoft.Dotnet.ILCompiler --prerelease
dotnet publish -r linux-x64 -c Release /p:StaticallyLinked=true --self-contained
./bin/Release/net7.0/linux-x64/native/HelloWorld

Expected behavior

Output "Hello, World!"

Actual behavior

Thread 1 "HelloWorld" received signal SIGSEGV, Segmentation fault.
0x00007fffd74b6603 in void std::call_once<void (&)()>(std::once_flag&, void (&)()) () from /lib/x86_64-linux-gnu/libicuuc.so.66
(gdb) bt
#0  0x00007fffd74b6603 in void std::call_once<void (&)()>(std::once_flag&, void (&)()) () from /lib/x86_64-linux-gnu/libicuuc.so.66
#1  0x00007fffd74b63b4 in icu_66::umtx_initImplPreInit(icu_66::UInitOnce&) () from /lib/x86_64-linux-gnu/libicuuc.so.66
#2  0x00007fffd74e79cb in ?? () from /lib/x86_64-linux-gnu/libicuuc.so.66
#3  0x00007fffd52c0a60 in ulocdata_getCLDRVersion_66 () from /lib/x86_64-linux-gnu/libicui18n.so.66
#4  0x0000000000476de2 in ValidateICUDataCanLoad () at /__w/1/s/src/native/libs/System.Globalization.Native/pal_icushim.c:375
#5  GlobalizationNative_LoadICU () at /__w/1/s/src/native/libs/System.Globalization.Native/pal_icushim.c:471
#6  0x0000000000633c5c in S_P_CoreLib_System_Globalization_GlobalizationMode_Settings___cctor ()
#7  0x00000000006209a9 in S_P_CoreLib_System_Runtime_CompilerServices_ClassConstructorRunner__EnsureClassConstructorRun ()
#8  0x00000000006208b9 in S_P_CoreLib_System_Runtime_CompilerServices_ClassConstructorRunner__CheckStaticClassConstructionReturnNonGCStaticBase ()
#9  0x00000000005cd103 in S_P_CoreLib_System_Globalization_CultureData__CreateCultureWithInvariantData ()
#10 0x00000000005cd1ca in S_P_CoreLib_System_Globalization_CultureData__get_Invariant ()
#11 0x00000000005ddf7c in S_P_CoreLib_System_Globalization_TextInfo___cctor ()
#12 0x00000000006209a9 in S_P_CoreLib_System_Runtime_CompilerServices_ClassConstructorRunner__EnsureClassConstructorRun ()
#13 0x00000000006208a9 in S_P_CoreLib_System_Runtime_CompilerServices_ClassConstructorRunner__CheckStaticClassConstructionReturnGCStaticBase ()
#14 0x000000000064cd17 in System_Console_System_Text_EncodingHelper__GetCharset ()
#15 0x000000000064cc29 in System_Console_System_Text_EncodingHelper__GetEncodingFromCharset ()
#16 0x0000000000645c09 in System_Console_System_ConsolePal__GetConsoleEncoding ()
#17 0x0000000000644cf4 in System_Console_System_Console__get_OutputEncoding ()
#18 0x0000000000644d9b in System_Console_System_Console__CreateOutputWriter ()
#19 0x0000000000644f43 in System_Console_System_Console___get_Out_g__EnsureInitialized_26_0 ()
#20 0x0000000000644e80 in System_Console_System_Console__WriteLine_12 ()
#21 0x00000000006e9f1f in __managed__Main ()
#22 0x00000000004058d9 in main (argc=1, argv=0x7fffffffe478) at /__w/1/s/src/coreclr/nativeaot/Bootstrap/main.cpp:205

Regression?

No response

Known Workarounds

Either disable globalization with <InvariantGlobalization>true</InvariantGlobalization> or don't statically link (remove /p:StaticallyLinked=true argument)

Configuration

dotnet version: 7.0.100-preview.3.22179.4

OS: Ubuntu 20.04.4 LTS

Architecture: x64

Other information

Reproduces on Ubuntu, but not AmazonLinux2. Ubuntu has libicuuc.so.66 but AmazonLinux2 has libicuuc.so.50

We can take a look at this, but were wondering if it was already known.

Author:	Beau-Gosse-dev
Assignees:	-
Labels:	area-System.Globalization
Milestone:	-

[smhmhmd]

There seem to be 2 versions of GlobalizationNative_LoadICU (pal_icushim.c, pal_icushim_static.c).
mono seems to be using the the static version.

Would it be okay to assign this issue to ourselves to use the static version of GlobalizationNative_LoadICU since this is a static build ?

If so, would you suggest using a new entrypoint for static like this in native/libs/System.Globalization.Native/entrypoints.c:
#if defined(STATIC_ICU)
DllImportEntry(GlobalizationNative_LoadICUData)
#endif

ubuntu@ip-172-31-24-113:~/runtime-net-7/runtime/src$ grep -r pal_icushim_static *
mono/wasm/runtime/es6/dotnet.es6.lib.js: // pal_icushim_static.c
mono/wasm/runtime/exports.ts: // also keep in sync with pal_icushim_static.c
mono/wasm/runtime/cjs/dotnet.cjs.lib.js: // pal_icushim_static.c
mono/wasm/runtime/icu.ts:// etc, see "mono_wasm_get_icudt_name" implementation in pal_icushim_static.c
mono/mono/mini/CMakeLists.txt: pal_icushim_static.c)

ubuntu@ip-172-31-24-113:~/runtime-net-7/runtime/src$ grep -r GlobalizationNative_LoadICU *
libraries/Common/src/Interop/Interop.ICU.iOS.cs: [LibraryImport(Libraries.GlobalizationNative, EntryPoint = "GlobalizationNative_LoadICUData", StringMarshalling = StringMarshalling.Utf8)]
libraries/Common/src/Interop/Interop.ICU.cs: [LibraryImport(Libraries.GlobalizationNative, EntryPoint = "GlobalizationNative_LoadICU")]
mono/wasi/mono-wasi-driver/driver.c:void GlobalizationNative_LoadICU() {
mono/wasi/mono-wasi-driver/driver.c: {"GlobalizationNative_LoadICU", GlobalizationNative_LoadICU },
native/libs/System.Globalization.Native/pal_icushim.c:// GlobalizationNative_LoadICU
native/libs/System.Globalization.Native/pal_icushim.c:int32_t GlobalizationNative_LoadICU()
native/libs/System.Globalization.Native/entrypoints.c: DllImportEntry(GlobalizationNative_LoadICU)
native/libs/System.Globalization.Native/entrypoints.c: DllImportEntry(GlobalizationNative_LoadICUData)
native/libs/System.Globalization.Native/pal_icushim_static.c:GlobalizationNative_LoadICUData(const char* path)
native/libs/System.Globalization.Native/pal_icushim_static.c: return GlobalizationNative_LoadICU();
native/libs/System.Globalization.Native/pal_icushim_static.c:int32_t GlobalizationNative_LoadICU(void)
native/libs/System.Globalization.Native/pal_icushim.h:PALEXPORT int32_t GlobalizationNative_LoadICU(void);
native/libs/System.Globalization.Native/pal_icushim.h:PALEXPORT int32_t GlobalizationNative_LoadICUData(const char* path);
tasks/AndroidAppBuilder/Templates/CMakeLists-android.txt: "-u GlobalizationNative_LoadICU"

[tarekgh]

CC @jkotas @MichalStrehovsky

[jkotas]

The full static linking is incompatible with any dynamic libraries being loaded in the process. That's how Linux and glibc work. You would need to create a custom build of the runtime, shims and dependencies that can be then all statically linked together and where nothing ever calls dlopen.

I am deleting the StaticallyLinked property in #70854 to avoid confusion. This property is has no chance of working well given how things are setup. If you want to experiment with static linking, you can just add -static linked arg to linked args in your local project file.

[smhmhmd]

@jkotas

If you want to experiment with static linking, you can just add -static linked arg to linked args in your local project file.

If a user chooses -static, can the static version of the GlobalizationNative_LoadICU() in pal_icushim_static.c be used instead of GlobalizationNative_LoadICU() in pal_icushim.c ?

[smhmhmd]

The full static linking is incompatible with any dynamic libraries being loaded in the process. That's how Linux and glibc work. You would need to create a custom build of the runtime, shims and dependencies that can be then all statically linked together and where nothing ever calls dlopen.

Static build compiles in Alpine Linux using libmusl, it did not show the dlopen issue in my test.
I did see the dlopen issue with glibc.

[jkotas]

can the static version of the GlobalizationNative_LoadICU() in pal_icushim_static.c be used instead of GlobalizationNative_LoadICU() in pal_icushim.c ?

You would need to do more than that to get statically linked ICU. For example, you would need to rebuild the whole System.Globalization.Native.a with STATIC_ICU defined.

[smhmhmd]

@jkotas

You would need to do more than that to get statically linked ICU

Yes, you are right.

Since libSystem.Globalization.Native.a is an archive, libicu*.a archives need to be linked separately like in the prototype below.
Here is a prototype to make the statically linked binary invoking ulocdata_getCLDRVersion() on Alpine Linux:
( pal*.o are from my dotnet build )

# cat /etc/os-release 
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.15.4
PRETTY_NAME="Alpine Linux v3.15"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
/container/tmp/x1 # 

/container/tmp/x1 # cat test.c
#include <stdio.h>
#include <unicode/uversion.h>
#include <unicode/errorcode.h>

int main()
{
        UVersionInfo version = {0};
        UErrorCode err = U_ZERO_ERROR;

        ulocdata_getCLDRVersion(version, &err);
	printf("Got CLDRVersion, err:%d\n", err);

        return 0;
}

# sh -x make.sh 
+ ar qc mypals.a pal_calendarData.c.o pal_icushim_static.c.o pal_localeNumberData.c.o pal_timeZoneInfo.c.o pal_casing.c.o pal_idna.c.o pal_localeStringData.c.o pal_collation.c.o pal_locale.c.o pal_normalization.c.o
+ gcc -c test.c
+ g++ -static -lstdc++ test.o mypals.a -Wl,--whole-archive /usr/lib/libicuio.a /usr/lib/libicutu.a /usr/lib/libicui18n.a /usr/lib/libicutest.a /usr/lib/libicuuc.a /usr/lib/libicudata.a -Wl,--no-whole-archive -o test
+ file test
test: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped
+ ./test
Got CLDRVersion, err:0

If you are okay with this approach, can I send a PR, assuming pal_icu_shim_static can be used under STATIC_ICU as an alternative to pal_icu_shim.c.
With clang-12 instead of gcc above, there is a warning, but the compilation flags dont error out:
warning: implicit declaration of function 'ulocdata_getCLDRVersion_69'