dotnet command hangs on two out of three Linux systems (possible deadlock in LTTNG)

[rubin55]

Describe the bug

Running dotnet without arguments shows the general help output. However, when trying to run dotnet publish or dotnet --info, or other dotnet commands, the command hangs indefinitely.

To Reproduce

I can reproduce this on two of my three Linux systems, with .NET core 6.x, 7.x and 8.x. I initially ran into this when trying to build netcoredbg and filed an issue with them here: Samsung/netcoredbg#162.

I later figured out that it was not with the netcoredbg project, but more fundamentally with dotnet itself. I found three issues which seem similar but I can't be sure:

• Dotnet 'myApp.dll' hang in lttng running on arm32 #9494
• dotnet tools occasionally hang  sdk#30455
• dotnet tools hanging when using SDK with Mono built runtime sdk#37774

Details of the machines I've tested this on (all running an identical setup of Arch Linux):

• Desktop at home (13900T with NVIDIA 4090 graphics, 128GiB ram): hangs
• Desktop at work (13900T with NVIDIA 3090 graphics, 128GiB ram): hangs
• Laptop (Framework Laptop 13, 1370P with Intel graphics, 64GiB ram): works

Exceptions (if any)

None, the tool hangs without any feedback, even when specifying verbose logging.

Further technical details

• Tested with the latest versions of dotnet 6.x, 7.x, 8.x
• on Arch Linux, with Linux kernel 6.7.4
• Core dump from hung process: core.zip

GDB Backtrace

futex_wait (private=0, expected=2, futex_word=0x7068751f7080) at ../sysdeps/nptl/futex-internal.h:146
146       int err = lll_futex_timed_wait (futex_word, expected, NULL, private);                                                     
(gdb) bt
#0  futex_wait (private=0, expected=2, futex_word=0x7068751f7080) at ../sysdeps/nptl/futex-internal.h:146
dotnet/sdk#1  __GI___lll_lock_wait (futex=futex@entry=0x7068751f7080, private=0) at lowlevellock.c:49
dotnet/sdk#2  0x0000706875aac9b2 in lll_mutex_lock_optimized (mutex=0x7068751f7080) at pthread_mutex_lock.c:48
dotnet/sdk#3  ___pthread_mutex_lock (mutex=0x7068751f7080) at pthread_mutex_lock.c:93
dotnet/sdk#4  0x000070687519799d in ust_lock_nocheck () from /usr/lib/liblttng-ust.so.0
dotnet/sdk#5  0x00007068751a743d in ?? () from /usr/lib/liblttng-ust.so.0
dotnet/sdk#6  0x00007068751a7572 in lttng_probe_register () from /usr/lib/liblttng-ust.so.0
dotnet/sdk#7  0x0000706875977383 in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.1/libcoreclrtraceptprovider.so
dotnet/sdk#8  0x00007068760bd1be in call_init (env=0x7ffd018e1e40, argv=0x7ffd018e1e28, argc=2, l=<optimized out>) at dl-init.c:74
dotnet/sdk#9  call_init (l=<optimized out>, argc=2, argv=0x7ffd018e1e28, env=0x7ffd018e1e40) at dl-init.c:26
dotnet/sdk#10 0x00007068760bd2ac in _dl_init (main_map=0x579486b15f30, argc=2, argv=0x7ffd018e1e28, env=0x7ffd018e1e40) at dl-init.c:121
dotnet/sdk#11 0x00007068760b956e in __GI__dl_catch_exception (exception=exception@entry=0x0, 
    operate=operate@entry=0x7068760c3d80 <call_dl_init>, args=args@entry=0x7ffd018e07f0) at dl-catch.c:211
dotnet/sdk#12 0x00007068760c3d26 in dl_open_worker (a=a@entry=0x7ffd018e0990) at dl-open.c:829
dotnet/sdk#13 0x00007068760b94e1 in __GI__dl_catch_exception (exception=exception@entry=0x7ffd018e0970, 
    operate=operate@entry=0x7068760c3c90 <dl_open_worker>, args=args@entry=0x7ffd018e0990) at dl-catch.c:237
dotnet/sdk#14 0x00007068760c410c in _dl_open (
    file=0x7ffd018e0c70 "/usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.1/libcoreclrtraceptprovider.so", mode=<optimized out>, 
    caller_dlopen=0x70687563f83b, nsid=<optimized out>, argc=2, argv=0x7ffd018e1e28, env=0x7ffd018e1e40) at dl-open.c:905
dotnet/sdk#15 0x0000706875aa53fc in dlopen_doit (a=a@entry=0x7ffd018e0c00) at dlopen.c:56
dotnet/sdk#16 0x00007068760b94e1 in __GI__dl_catch_exception (exception=exception@entry=0x7ffd018e0b60, 
    operate=0x706875aa53a0 <dlopen_doit>, args=0x7ffd018e0c00) at dl-catch.c:237
dotnet/sdk#17 0x00007068760b9603 in _dl_catch_error (objname=0x7ffd018e0bb8, errstring=0x7ffd018e0bc0, mallocedp=0x7ffd018e0bb7, 
    operate=<optimized out>, args=<optimized out>) at dl-catch.c:256
dotnet/sdk#18 0x0000706875aa4f07 in _dlerror_run (operate=operate@entry=0x706875aa53a0 <dlopen_doit>, args=args@entry=0x7ffd018e0c00)
    at dlerror.c:138
dotnet/sdk#19 0x0000706875aa54b1 in dlopen_implementation (dl_caller=<optimized out>, mode=<optimized out>, file=<optimized out>)
    at dlopen.c:71
dotnet/sdk#20 ___dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:81
dotnet/sdk#21 0x000070687563f83b in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.1/libcoreclr.so
dotnet/sdk#22 0x00007068760bd1be in call_init (env=0x7ffd018e1e40, argv=0x7ffd018e1e28, argc=2, l=<optimized out>) at dl-init.c:74
dotnet/sdk#23 call_init (l=<optimized out>, argc=2, argv=0x7ffd018e1e28, env=0x7ffd018e1e40) at dl-init.c:26
dotnet/sdk#24 0x00007068760bd2ac in _dl_init (main_map=0x579486b12ae0, argc=2, argv=0x7ffd018e1e28, env=0x7ffd018e1e40) at dl-init.c:121
dotnet/sdk#25 0x00007068760b956e in __GI__dl_catch_exception (exception=exception@entry=0x0, 
    operate=operate@entry=0x7068760c3d80 <call_dl_init>, args=args@entry=0x7ffd018e0f60) at dl-catch.c:211
dotnet/sdk#26 0x00007068760c3d26 in dl_open_worker (a=a@entry=0x7ffd018e1100) at dl-open.c:829
dotnet/sdk#27 0x00007068760b94e1 in __GI__dl_catch_exception (exception=exception@entry=0x7ffd018e10e0, 
    operate=operate@entry=0x7068760c3c90 <dl_open_worker>, args=args@entry=0x7ffd018e1100) at dl-catch.c:237
dotnet/sdk#28 0x00007068760c410c in _dl_open (file=0x579486b263d0 "/usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.1/libcoreclr.so", 
    mode=<optimized out>, caller_dlopen=0x706875ee6c1d, nsid=<optimized out>, argc=2, argv=0x7ffd018e1e28, env=0x7ffd018e1e40)
    at dl-open.c:905
dotnet/sdk#29 0x0000706875aa53fc in dlopen_doit (a=a@entry=0x7ffd018e1370) at dlopen.c:56
dotnet/sdk#30 0x00007068760b94e1 in __GI__dl_catch_exception (exception=exception@entry=0x7ffd018e12d0, 
    operate=0x706875aa53a0 <dlopen_doit>, args=0x7ffd018e1370) at dl-catch.c:237
dotnet/sdk#31 0x00007068760b9603 in _dl_catch_error (objname=0x7ffd018e1328, errstring=0x7ffd018e1330, mallocedp=0x7ffd018e1327, 
    operate=<optimized out>, args=<optimized out>) at dl-catch.c:256
dotnet/sdk#32 0x0000706875aa4f07 in _dlerror_run (operate=operate@entry=0x706875aa53a0 <dlopen_doit>, args=args@entry=0x7ffd018e1370)
    at dlerror.c:138
dotnet/sdk#33 0x0000706875aa54b1 in dlopen_implementation (dl_caller=<optimized out>, mode=<optimized out>, file=<optimized out>)
    at dlopen.c:71
dotnet/sdk#34 ___dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:81
dotnet/sdk#35 0x0000706875ee6c1d in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.1/libhostpolicy.so
dotnet/sdk#36 0x0000706875ebbc53 in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.1/libhostpolicy.so
dotnet/sdk#37 0x0000706875ebd069 in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.1/libhostpolicy.so
dotnet/sdk#38 0x0000706875ed9581 in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.1/libhostpolicy.so
dotnet/sdk#39 0x0000706875ed8f94 in corehost_main () from /usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.1/libhostpolicy.so
dotnet/sdk#40 0x0000706875f150aa in ?? () from /usr/share/dotnet/host/fxr/8.0.1/libhostfxr.so
dotnet/sdk#41 0x0000706875f147bc in ?? () from /usr/share/dotnet/host/fxr/8.0.1/libhostfxr.so
dotnet/sdk#42 0x0000706875f1405f in ?? () from /usr/share/dotnet/host/fxr/8.0.1/libhostfxr.so
dotnet/sdk#43 0x0000706875f101e2 in hostfxr_main_startupinfo () from /usr/share/dotnet/host/fxr/8.0.1/libhostfxr.so
dotnet/sdk#44 0x0000579485ba1d88 in ?? ()
dotnet/sdk#45 0x0000579485ba206f in ?? ()
dotnet/sdk#46 0x0000706875a43cd0 in __libc_start_call_main (main=main@entry=0x579485ba1fc0, argc=argc@entry=2, 
    argv=argv@entry=0x7ffd018e1e28) at ../sysdeps/nptl/libc_start_call_main.h:58
dotnet/sdk#47 0x0000706875a43d8a in __libc_start_main_impl (main=0x579485ba1fc0, argc=2, argv=0x7ffd018e1e28, init=<optimized out>, 
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd018e1e18) at ../csu/libc-start.c:360
dotnet/sdk#48 0x0000579485b98645 in ?? ()

[tommcdon]

@davmason

[davmason]

@brianrob do you recognize the callstack above? is this another symptom of the LTTng breaking abi changes?

[brianrob]

I don't expect this to be caused by the breaking abi change because in that case, dlopen would return NULL because the expected lttng-ust binary doesn't exist on disk (it's been renamed on purpose).

This looks familiar, but it's been a long time since I've seen this. It reminds me of a bug that existed in LTTng years back that was due to a difference in struct packing behavior between gdb and g++. I did a search of my e-mail, but I could not find it.

What version of LTTng is installed on this machine?

[rubin55]

What version of LTTng is installed on this machine?

This is an up-to-date Arch Linux installation: lttng-ust2.12 2.12.9-1

[tommcdon]

This is an up-to-date Arch Linux installation: lttng-ust2.12 2.12.9-1

@brianrob lttng 2.12 looks like a recent version. Do you have other thoughts, perhaps an incompatibility between lttng and Arch linux?
@rubin55 Out of curiosity does this issue reproduce on one of our supported OS's? https://github.com/dotnet/core/blob/main/release-notes/9.0/supported-os.md

[brianrob]

@tommcdon agreed that is a new version. I doubt that they're incompatible, but here's the installation and compatibility info for LTTng 2.12: https://lttng.org/docs/v2.12/#doc-alpine-linux. I don't see the version of Arch Linux here (or I missed it), so it would be good to confirm that the version is listed on the LTTng link.

Do you know what version of the LTTng headers we're building with? I wonder if it's possible that there is an unintended bug there that perhaps is fixed in a newer version of the headers. I have no proof of this - just a thought.

[MichalPetryka]

the version of Arch Linux

Arch Linux is a non versioned distro, the only somewhat version like thing is creation dates of installer images.

[brianrob]

Sorry, for some reason, I read Arch as Alpine. In that case, yes it's possible that there are issues. I don't see anything about installation using Arch Linux, but I would imagine that you're building from source.