Infinite recursion during resource lookup within System.Private.CoreLib

[fanyang-mono]

Steps to Reproduce on linux

1. Build mono-netcore with llvm JIT: ./build.sh -c Release /p:MonoEnableLLVM=true
2. Get TE benchmark: git clone https://github.com/aspnet/Benchmarks.git
3. Build benchmark:

export BenchmarksTargetFramework=netcoreapp5.0
export MicrosoftAspNetCoreAppPackageVersion=5.0.0-alpha.1.20071.6
export MicrosoftNETCoreAppPackageVersion=5.0.0-alpha.1.20070.4
.dotnet/dotnet publish -c Release -f netcoreapp5.0 --self-contained -r linux-x64 <TE_benchmark_git_root>/src/BenchmarksApps/Kestrel/PlatformBenchmarks
cp artifacts/obj/mono/Linux.x64.Release/mono/mini/.libs/libmonosgen-2.0.so <TE_benchmark_git_root>/src/BenchmarksApps/Kestrel/PlatformBenchmarks/bin/Release/netcoreapp5.0/linux-x64/publish/libcoreclr.so
cp artifacts/bin/mono/Linux.x64.Release/System.Private.CoreLib.dll <TE_benchmark_git_root>/src/BenchmarksApps/Kestrel/PlatformBenchmarks/bin/Release/netcoreapp5.0/linux-x64/publish/

4. Run the app:

cd <TE_benchmark_git_root>/src/BenchmarksApps/Kestrel/PlatformBenchmarks/bin/Release/netcoreapp5.0/linux-x64/publish
export MONO_ENV_OPTIONS="--llvm --server --gc=sgen --gc-params=mode=throughput"
./PlatformBenchmarks

Current Behavior

CLR: Managed code called FailFast, saying "Infinite recursion during resource lookup within System.Private.CoreLib. This may be a bug in System.Private.CoreLib, or potentially in certain extensibility points such as assembly resolve events or CultureInfo names. Resource name: Arg_PlatformNotSupported"

Expected Behavior

No crash

On which platforms did you notice this

[x] macOS
[x] Linux
[ ] Windows

Version Used:
dotnet/runtime 13c1e65

Stacktrace

CLR: Managed code called FailFast, saying "Infinite recursion during resource lookup within System.Private.CoreLib.  This may be a bug in System.Private.CoreLib, or potentially in certain extensibility points such as assembly resolve events or CultureInfo names.  Resource name: Arg_PlatformNotSupported"

Thread 4 (Thread 0x7fa2057ff700 (LWP 31212)):
#0  0x00007fa21032b6d6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7fa20eef35b8 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x7fa20eef35b8 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007fa21032b7c8 in __new_sem_wait_slow (sem=0x7fa20eef35b8 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007fa20d69db98 in mono_os_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /home/parallels/Documents/runtime_latest/runtime/src/mono/mono/utils/mono-os-semaphore.h:203
#4  mono_coop_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /home/parallels/Documents/runtime_latest/runtime/src/mono/mono/utils/mono-coop-semaphore.h:41
#5  finalizer_thread (unused=<optimized out>) at /home/parallels/Documents/runtime_latest/runtime/src/mono/mono/metadata/gc.c:965
#6  0x00007fa20d6578fa in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /home/parallels/Documents/runtime_latest/runtime/src/mono/mono/metadata/threads.c:1232
#7  0x00007fa20d657779 in start_wrapper (data=0x559f18907d30) at /home/parallels/Documents/runtime_latest/runtime/src/mono/mono/metadata/threads.c:1307
#8  0x00007fa2103226db in start_thread (arg=0x7fa2057ff700) at pthread_create.c:463
#9  0x00007fa20f50888f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7fa209402700 (LWP 31211)):
#0  0x00007fa2103289f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa20ef01bb8 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa20ef01b68 <lock>, cond=0x7fa20ef01b90 <work_cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fa20ef01b90 <work_cond>, mutex=0x7fa20ef01b68 <lock>) at pthread_cond_wait.c:655
#3  0x00007fa20d6ef793 in mono_os_cond_wait (cond=<optimized out>, mutex=<optimized out>) at /home/parallels/Documents/runtime_latest/runtime/src/mono/mono/utils/mono-os-mutex.h:219
#4  get_work (worker_index=<optimized out>, work_context=<optimized out>, do_idle=<optimized out>, job=<optimized out>) at /home/parallels/Documents/runtime_latest/runtime/src/mono/mono/sgen/sgen-thread-pool.c:165
#5  thread_func (data=0x1) at /home/parallels/Documents/runtime_latest/runtime/src/mono/mono/sgen/sgen-thread-pool.c:196
#6  0x00007fa2103226db in start_thread (arg=0x7fa209402700) at pthread_create.c:463
#7  0x00007fa20f50888f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7fa209c03700 (LWP 31210)):
#0  0x00007fa2103289f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa20ef01bb8 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa20ef01b68 <lock>, cond=0x7fa20ef01b90 <work_cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fa20ef01b90 <work_cond>, mutex=0x7fa20ef01b68 <lock>) at pthread_cond_wait.c:655
#3  0x00007fa20d6ef793 in mono_os_cond_wait (cond=<optimized out>, mutex=<optimized out>) at /home/parallels/Documents/runtime_latest/runtime/src/mono/mono/utils/mono-os-mutex.h:219
#4  get_work (worker_index=<optimized out>, work_context=<optimized out>, do_idle=<optimized out>, job=<optimized out>) at /home/parallels/Documents/runtime_latest/runtime/src/mono/mono/sgen/sgen-thread-pool.c:165
#5  thread_func (data=0x0) at /home/parallels/Documents/runtime_latest/runtime/src/mono/mono/sgen/sgen-thread-pool.c:196
#6  0x00007fa2103226db in start_thread (arg=0x7fa209c03700) at pthread_create.c:463
#7  0x00007fa20f50888f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7fa210742740 (LWP 31209)):
#0  0x00007fa21032d23a in __waitpid (pid=31220, stat_loc=0x7ffdd4a83b7c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x00007fa20d566e21 in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /home/parallels/Documents/runtime_latest/runtime/src/mono/mono/mini/mini-posix.c:1113
#2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0x7ffdd4a84690, info=<optimized out>) at /home/parallels/Documents/runtime_latest/runtime/src/mono/mono/mini/mini-posix.c:1157
#3  0x00007fa20d4e6f35 in mono_handle_native_crash (signal=0x7fa20e5fa0fc "SIGABRT", mctx=0x7ffdd4a84690, info=0x7ffdd4a84970) at /home/parallels/Documents/runtime_latest/runtime/src/mono/mono/mini/mini-exceptions.c:3427
#4  0x00007fa20d566275 in sigabrt_signal_handler (_dummy=6, _info=0x7ffdd4a84970, context=0x7ffdd4a84840) at /home/parallels/Documents/runtime_latest/runtime/src/mono/mono/mini/mini-posix.c:234
#5  <signal handler called>
#6  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#7  0x00007fa20f427801 in __GI_abort () at abort.c:79
#8  0x00007fa20d5f94d4 in ves_icall_System_Environment_FailFast (message=..., exception=..., errorSource=..., error=<optimized out>) at /home/parallels/Documents/runtime_latest/runtime/src/mono/mono/metadata/icall.c:7760
#9  0x00007fa20d5fbc04 in ves_icall_System_Environment_FailFast_raw (a0=0x7ffdd4a85220, a1=0x7ffdd4a85228, a2=<optimized out>) at /home/parallels/Documents/runtime_latest/runtime/src/mono/mono/metadata/icall-def-netcore.h:79
#10 0x0000000041774615 in ?? ()
#11 0x00007fa206006d80 in ?? ()
#12 0x00007ffdd4a855f0 in ?? ()
#13 0x0000559f18a2e478 in ?? ()
#14 0x00007fa208402000 in ?? ()
#15 0x00007fa208402000 in ?? ()
#16 0x0000559f18899910 in ?? ()
#17 0x00007ffdd4a85320 in ?? ()
#18 0x00007ffdd4a851e0 in ?? ()
#19 0x00007fa206007030 in ?? ()
#20 0x0000000000000000 in ?? ()

[CoffeeFlux]

From Slack:

This happens when you're missing System.Globalization.Native.dylib. You can get rid of it by either building libraries or setting DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1.

[marek-safar]

/cc @steveisok

[fanyang-mono]

From Slack:

This happens when you're missing System.Globalization.Native.dylib. You can get rid of it by either building libraries or setting DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1.

This doesn't seem to solve this problem.

[steveisok]

I can reproduce. And what's weird is that DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1 does not help.

It's important to point out that this all works normal when you don't run under --llvm. I do wonder if there's a loader problem since the exception we're getting suggests that "potentially dangerous" code that throws will cause an infinite loop.

Here's the last bit of the mono trace before it bombs:

Mono: Assembly Loader probing location: '/Users/steve/dev/net5/Benchmarks/src/BenchmarksApps/Kestrel/PlatformBenchmarks/bin/Release/netcoreapp5.0/osx-x64/publish/System.Runtime.Extensions.dll'.
Mono: Image addref System.Runtime.Extensions[0x7fc9bcc0a100] (asmctx DEFAULT) -> /Users/steve/dev/net5/Benchmarks/src/BenchmarksApps/Kestrel/PlatformBenchmarks/bin/Release/netcoreapp5.0/osx-x64/publish/System.Runtime.Extensions.dll[0x7fc9cd802200]: 2
Mono: Predicate: wanted = System.Runtime.Extensions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
Mono: Predicate: candidate = System.Runtime.Extensions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
Mono: Predicate: candidate and wanted names match, returning TRUE
Mono: Prepared to set up assembly 'System.Runtime.Extensions' (/Users/steve/dev/net5/Benchmarks/src/BenchmarksApps/Kestrel/PlatformBenchmarks/bin/Release/netcoreapp5.0/osx-x64/publish/System.Runtime.Extensions.dll)
Mono: Loading assembly System.Runtime.Extensions (0x7fc9bcc0a100) into domain PlatformBenchmarks.dll (0x7fc9acd0fcc0) and ALC 0x7fc9acd10730
Mono: Assembly System.Runtime.Extensions[0x7fc9bcc0a100] added to domain PlatformBenchmarks.dll, ref_count=1
Mono: Assembly System.Runtime.Extensions[0x7fc9bcc0a100] added to ALC (0x7fc9acd10730), ref_count=2
Mono: AOT: image '/Users/steve/dev/net5/Benchmarks/src/BenchmarksApps/Kestrel/PlatformBenchmarks/bin/Release/netcoreapp5.0/osx-x64/publish/System.Runtime.Extensions.dll.dylib' not found: dlopen(/Users/steve/dev/net5/Benchmarks/src/BenchmarksApps/Kestrel/PlatformBenchmarks/bin/Release/netcoreapp5.0/osx-x64/publish/System.Runtime.Extensions.dll.dylib, 1): image not found
Mono: AOT: image '/usr/local/lib/mono/aot-cache/amd64/System.Runtime.Extensions.dll.dylib' not found: dlopen(/usr/local/lib/mono/aot-cache/amd64/System.Runtime.Extensions.dll.dylib, 1): image not found
Mono: Assembly Loader loaded assembly from location: '/Users/steve/dev/net5/Benchmarks/src/BenchmarksApps/Kestrel/PlatformBenchmarks/bin/Release/netcoreapp5.0/osx-x64/publish/System.Runtime.Extensions.dll'.
Mono: Config attempting to parse: '/Users/steve/dev/net5/Benchmarks/src/BenchmarksApps/Kestrel/PlatformBenchmarks/bin/Release/netcoreapp5.0/osx-x64/publish/System.Runtime.Extensions.dll.config'.
Mono: Config attempting to parse: '/usr/local/etc/mono/assemblies/System.Runtime.Extensions/System.Runtime.Extensions.config'.
Mono: Assembly Ref addref System.Console[0x7fc9bcc090f0] -> System.Runtime.Extensions[0x7fc9bcc0a100]: 3
Mono: Requesting loading reference 0 (of 2) of /Users/steve/dev/net5/Benchmarks/src/BenchmarksApps/Kestrel/PlatformBenchmarks/bin/Release/netcoreapp5.0/osx-x64/publish/System.Runtime.Extensions.dll
Mono: Loading reference 0 of /Users/steve/dev/net5/Benchmarks/src/BenchmarksApps/Kestrel/PlatformBenchmarks/bin/Release/netcoreapp5.0/osx-x64/publish/System.Runtime.Extensions.dll asmctx DEFAULT, looking for System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e
Mono: Request to load System.Private.CoreLib in (domain 0x7fc9acd0fcc0, alc 0x7fc9acd10730)
Mono: Assembly Ref addref System.Runtime.Extensions[0x7fc9bcc0a100] -> System.Private.CoreLib[0x7fc9acd13200]: 4
CLR: Managed code called FailFast, saying "Infinite recursion during resource lookup within System.Private.CoreLib.  This may be a bug in System.Private.CoreLib, or potentially in certain extensibility points such as assembly resolve events or CultureInfo names.  Resource name: Arg_PlatformNotSupported"

/cc @lambdageek

[marek-safar]

It could be a few issues mixed together. DOTNET_SYSTEM_GLOBALIZATION_INVARIANT requires some hook to be ready very early. We might be not accomodating for that and it's worth checking if it works at all.

There is also some ugly code in https://github.com/dotnet/runtime/blob/master/src/libraries/System.Private.CoreLib/src/System/SR.cs#L95 which might be CoreCLR specific.

/cc @lambdageek

[steveisok]

I did another pass with and without DOTNET_SYSTEM_GLOBALIZATION_INVARIANT. With the setting I do not see a trace that loads System.Globalization.Native, but without I do.

Without:

Mono: DllImport searching in: 'System.Globalization.Native' ('System.Globalization.Native.dylib').
Mono: Searching for 'GlobalizationNative_GetLocaleName'.
Mono: Probing 'GlobalizationNative_GetLocaleNameW'.
Mono: Could not find 'GlobalizationNative_GetLocaleNameW' due to 'dlsym(0x7fee10603c30, GlobalizationNative_GetLocaleNameW): symbol not found'.
Mono: Probing 'GlobalizationNative_GetLocaleName'.
Mono: Found as 'GlobalizationNative_GetLocaleName'.
Mono: DllImport attempting to load: 'System.Globalization.Native'.
Mono: DllImport searching in: 'System.Globalization.Native' ('System.Globalization.Native.dylib').
Mono: Searching for 'GlobalizationNative_GetSortHandle'.
Mono: Probing 'GlobalizationNative_GetSortHandle'.
Mono: Found as 'GlobalizationNative_GetSortHandle'.
Mono: DllImport attempting to load: 'System.Globalization.Native'.
Mono: DllImport searching in: 'System.Globalization.Native' ('System.Globalization.Native.dylib').
Mono: Searching for 'GlobalizationNative_CloseSortHandle'.
Mono: Probing 'GlobalizationNative_CloseSortHandle'.
Mono: Found as 'GlobalizationNative_CloseSortHandle'.
Mono: DllImport attempting to load: 'System.Globalization.Native'.
Mono: DllImport searching in: 'System.Globalization.Native' ('System.Globalization.Native.dylib').
Mono: Searching for 'GlobalizationNative_GetLocaleInfoString'.
Mono: Probing 'GlobalizationNative_GetLocaleInfoStringW'.
Mono: Could not find 'GlobalizationNative_GetLocaleInfoStringW' due to 'dlsym(0x7fee10603c30, GlobalizationNative_GetLocaleInfoStringW): symbol not found'.
Mono: Probing 'GlobalizationNative_GetLocaleInfoString'.
Mono: Found as 'GlobalizationNative_GetLocaleInfoString'.
CLR: Managed code called FailFast, saying "Infinite recursion during resource lookup within System.Private.CoreLib.  This may be a bug in System.Private.CoreLib, or potentially in certain extensibility points such as assembly resolve events or CultureInfo names.  Resource name: Arg_PlatformNotSupported"

[marek-safar]

Could you paste the log with the setting of the env variable?

[steveisok]

https://gist.github.com/steveisok/82454f8fcbaef105084b8bbeae761000