[mono][metadata] Replace use of mem manager lock with loader lock

[BrzVlad]

Hash table operations under the mem manager lock could end up taking the loader lock when performing type comparison, in the case where custom modifiers needed to be loaded. Use the loader lock instead to prevent deadlocks.

Fixes #90834

[lambdageek]

We should backport to net8.0 /cc @SamMonoRT

[SamMonoRT]

/azp run runtime-extra-platforms

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[carlossanlop]

@BrzVlad @SamMonoRT @lambdageek is this ready to get backported? I think this is the most impactful failure in 8.0.

[BrzVlad]

/backport to release/8.0

[github-actions]

Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/6024778897

[vargaz]

Wouldn't be better to modify the hashtable equality functions so they wouldn't lock ?

[lambdageek]

That might be difficult - I think the issue is that they indirectly trigger assembly class loading. Although maybe there's some way to avoid it

[vargaz]

Is there a stacktrace which shows this happening ?

[BrzVlad]

    frame #0: 0x00007ff803797892 libsystem_kernel.dylib`__psynch_mutexwait + 10
    frame #1: 0x00007ff8037d1b14 libsystem_pthread.dylib`_pthread_mutex_firstfit_lock_wait + 76
    frame #2: 0x00007ff8037cf921 libsystem_pthread.dylib`_pthread_mutex_firstfit_lock_slow + 214
    frame #3: 0x000000010b1ff905 libcoreclr.dylib`mono_os_mutex_lock(mutex=0x000000010b3d8ec0) at mono-os-mutex.h:105:8
    frame #4: 0x000000010b1ffae1 libcoreclr.dylib`mono_coop_mutex_lock(mutex=0x000000010b3d8ec0) at mono-coop-mutex.h:57:2
    frame #5: 0x000000010b1ffa39 libcoreclr.dylib`mono_loader_lock at loader.c:136:2
    frame #6: 0x000000010b1ab4e8 libcoreclr.dylib`mono_class_create_from_typedef(image=0x00007f7b2f815200, type_token=33555983, error=0x0000700011720a80) at class-init.c:486:2
    frame #7: 0x000000010b19e7f1 libcoreclr.dylib`mono_class_get_checked(image=0x00007f7b2f815200, type_token=33555983, error=0x0000700011720a80) at class.c:2925:11
    frame #8: 0x000000010b19fc83 libcoreclr.dylib`mono_class_from_name_checked_aux(image=0x00007f7b2f815200, name_space="System.Runtime.CompilerServices", name="IsConst", visited_images=0x0000600000ffe5c0, case_sensitive=1, error=0x0000700011720a80) at class.c:3444:10
    frame #9: 0x000000010b19fbc7 libcoreclr.dylib`mono_class_from_name_checked_aux(image=0x00007f7b2e816200, name_space="System.Runtime.CompilerServices", name="IsConst", visited_images=0x0000600000ffe5c0, case_sensitive=1, error=0x0000700011720a80) at class.c:3433:12
    frame #10: 0x000000010b19758e libcoreclr.dylib`mono_class_from_name_checked(image=0x00007f7b2e816200, name_space="System.Runtime.CompilerServices", name="IsConst", error=0x0000700011720a80) at class.c:3470:10
    frame #11: 0x000000010b197456 libcoreclr.dylib`mono_class_from_typeref_checked(image=0x00007f7b2f902c00, type_token=16777219, error=0x0000700011720a80) at class.c:214:8
    frame #12: 0x000000010b19e80a libcoreclr.dylib`mono_class_get_checked(image=0x00007f7b2f902c00, type_token=16777219, error=0x0000700011720a80) at class.c:2928:11
    frame #13: 0x000000010b19eae1 libcoreclr.dylib`mono_type_get_checked(image=0x00007f7b2f902c00, type_token=16777219, context=0x0000000000000000, error=0x0000700011720a80) at class.c:2976:22
    frame #14: 0x000000010b226dd8 libcoreclr.dylib`mono_type_get_custom_modifier(ty=0x00006000014fdb90, idx='\0', required=0x0000700011720a74, error=0x0000700011720a80) at metadata.c:6043:10
    frame #15: 0x000000010b22dd5e libcoreclr.dylib`mono_metadata_custom_modifiers_equal(t1=0x00006000014fdb90, t2=0x00006000014e1110, signature_only=0) at metadata.c:5781:24
    frame #16: 0x000000010b2265b6 libcoreclr.dylib`do_mono_metadata_type_equal(t1=0x00006000014fdb90, t2=0x00006000014e1110, equiv_flags=0) at metadata.c:5816:19
    frame #17: 0x000000010b21f7b1 libcoreclr.dylib`mono_generic_inst_equal_full(a=0x00007f7b32030410, b=0x0000700011720e90, signature_only=0) at metadata.c:1909:8
    frame #18: 0x000000010b21f69f libcoreclr.dylib`mono_metadata_generic_inst_equal(ka=0x00007f7b32030410, kb=0x0000700011720e90) at metadata.c:1921:9
    frame #19: 0x000000010b14f53d libcoreclr.dylib`monoeg_g_hash_table_lookup_extended(hash=0x0000600000fcc740, key=0x0000700011720e90, orig_key=0x0000700011720c10, value=0x0000700011720c08) at ghashtable.c:341:7
    frame #20: 0x000000010b14f5d5 libcoreclr.dylib`monoeg_g_hash_table_lookup(hash=0x0000600000fcc740, key=0x0000700011720e90) at ghashtable.c:321:6
    frame #21: 0x000000010b221e46 libcoreclr.dylib`mono_metadata_get_canonical_generic_inst(candidate=0x0000700011720e90) at metadata.c:3442:46
    frame #22: 0x000000010b221ce8 libcoreclr.dylib`mono_metadata_get_generic_inst(type_argc=2, type_argv=0x00006000018bd260) at metadata.c:3409:9
    frame #23: 0x000000010b222c9f libcoreclr.dylib`mono_metadata_parse_generic_inst(m=0x00007f7b2f902c00, container=0x0000000000000000, count=2, ptr="\b0\U00000001\U00000001\U00000001 \r\U0000001e", rptr=0x0000700011720fa8, error=0x0000700011721360) at metadata.c:3635:10
    frame #24: 0x000000010b22cef3 libcoreclr.dylib`do_mono_metadata_parse_generic_class(type=0x00007000117210f8, m=0x00007f7b2f902c00, container=0x0000000000000000, ptr="\b0\U00000001\U00000001\U00000001 \r\U0000001e", rptr=0x0000700011721020, error=0x0000700011721360) at metadata.c:3668:9
    frame #25: 0x000000010b22c988 libcoreclr.dylib`do_mono_metadata_parse_type(type=0x00007000117210f8, m=0x00007f7b2f902c00, container=0x0000000000000000, transient=0, ptr="\U00000012\t\U00000002\b \r\U00000002\b0\U00000001\U00000001\U00000001 \r\U0000001e", rptr=0x00007000117210c0, error=0x0000700011721360) at metadata.c:4093:8
    frame #26: 0x000000010b22c2e9 libcoreclr.dylib`do_mono_metadata_parse_type_with_cmods(type=0x00007000117210f8, cmod_count=0, m=0x00007f7b2f902c00, container=0x0000000000000000, opt_attrs=0, transient=0, ptr="\U00000015\U00000012\t\U00000002\b \r\U00000002\b0\U00000001\U00000001\U00000001 \r\U0000001e", rptr=0x0000700011721208, error=0x0000700011721360) at metadata.c:2151:7
    frame #27: 0x000000010b21fb38 libcoreclr.dylib`mono_metadata_parse_type_internal(m=0x00007f7b2f902c00, container=0x0000000000000000, opt_attrs=0, transient=0, ptr="\U00000015\U00000012\t\U00000002\b \r\U00000002\b0\U00000001\U00000001\U00000001 \r\U0000001e", rptr=0x0000700011721208, error=0x0000700011721360) at metadata.c:2233:7
    frame #28: 0x000000010b21fa35 libcoreclr.dylib`mono_metadata_parse_type_checked(m=0x00007f7b2f902c00, container=0x0000000000000000, opt_attrs=0, transient=0, ptr="\U00000015\U00000012\t\U00000002\b \r\U00000002\b0\U00000001\U00000001\U00000001 \r\U0000001e", rptr=0x0000700011721208, error=0x0000700011721360) at metadata.c:2265:9
    frame #29: 0x000000010b2205e6 libcoreclr.dylib`mono_metadata_parse_method_signature_full(m=0x00007f7b2f902c00, container=0x0000000000000000, def=2, ptr="\U00000015\U00000012\t\U00000002\b \r\U00000002\b0\U00000001\U00000001\U00000001 \r\U0000001e", rptr=0x0000000000000000, error=0x0000700011721360) at metadata.c:2717:24
    frame #30: 0x000000010b2033ce libcoreclr.dylib`mono_method_signature_checked_slow(m=0x00007f7b3030ddf8, error=0x0000700011721360) at loader.c:1850:15
    frame #31: 0x000000010b200c4b libcoreclr.dylib`mono_method_signature_checked(m=0x00007f7b3030ddf8, error=0x0000700011721360) at class-internals.h:775:21
    frame #32: 0x000000010b20383b libcoreclr.dylib`mono_method_signature_internal_slow(m=0x00007f7b3030ddf8) at loader.c:1940:29
    frame #33: 0x000000010b1d0f09 libcoreclr.dylib`mono_method_signature_internal(m=0x00007f7b3030ddf8) at class-internals.h:786:21
    frame #34: 0x000000010b1d7cb1 libcoreclr.dylib`ves_icall_RuntimeMethodInfo_get_IsGenericMethod(ref_method=MonoReflectionMethodHandle @ 0x0000700011721418, erro=0x0000700011721460) at icall.c:3466:9
    frame #35: 0x000000010b1eb58b libcoreclr.dylib`ves_icall_RuntimeMethodInfo_get_IsGenericMethod_raw(a0=0x0000700011721520) at icall-def.h:402:1
    frame #36: 0x0000000110030708
    frame #37: 0x000000010ae3c273 libcoreclr.dylib`mono_jit_runtime_invoke(method=0x00007f7b2ebee728, obj=0x0000000000000000, params=0x0000000000000000, exc=0x00007000117216c0, error=0x0000700011721a50) at mini-runtime.c:3681:12
    frame #38: 0x000000010b23d3b6 libcoreclr.dylib`do_runtime_invoke(method=0x00007f7b2ebee728, obj=0x0000000000000000, params=0x0000000000000000, exc=0x0000000000000000, error=0x0000700011721a50) at object.c:2542:11
    frame #39: 0x000000010b237761 libcoreclr.dylib`mono_runtime_invoke_checked(method=0x00007f7b2ebee728, obj=0x0000000000000000, params=0x0000000000000000, error=0x0000700011721a50) at object.c:2758:9
    frame #40: 0x000000010b243bdf libcoreclr.dylib`mono_runtime_try_invoke_byrefs(method=0x00007f7b2ebee728, obj=0x0000000000000000, params_byref=0x0000000000000000, exc=0x0000000000000000, error=0x0000700011721a50) at object.c:5142:16
    frame #41: 0x000000010b1d8c14 libcoreclr.dylib`ves_icall_InternalInvoke(method_handle=MonoReflectionMethodHandle @ 0x0000700011721968, this_arg_handle=MonoObjectHandle @ 0x0000700011721960, params_byref=0x0000000000000000, exception_out=MonoExceptionHandleOut @ 0x0000700011721958, error=0x0000700011721a50) at icall.c:3622:11
    frame #42: 0x000000010b1ea48f libcoreclr.dylib`ves_icall_InternalInvoke_raw(a0=0x0000700011721b48, a1=0x0000700011721b50, a2=0x0000000000000000, a3=0x0000700011721b40) at icall-def.h:371:1