CI: Threads tests segfault on Julia nightly

[giordano]

Unrelated to this PR, something is horribly broken on master (Julia Version 1.12.0-DEV.1641 Commit 7fa26f011ec (2024-11-16 19:20 UTC)) with OpenMPI_jll:

malloc(): unaligned tcache chunk detected

[3920] signal 6 (-6): Aborted
in expression starting at /home/runner/work/MPI.jl/MPI.jl/test/test_threads.jl:18

[3920] signal 11 (1): Segmentation fault
in expression starting at /home/runner/work/MPI.jl/MPI.jl/test/test_threads.jl:18

[3918] signal 15: Terminated
in expression starting at /home/runner/work/MPI.jl/MPI.jl/test/test_threads.jl:39
unknown function (ip: 0x7f63d4691115) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/builder-amdci5-5/julialang/julia-master/src/gc-stock.c:3544
unknown function (ip: 0x7f63d4694ac2) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7f63d472684f) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7f63d4691115) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/builder-amdci5-5/julialang/julia-master/src/gc-stock.c:3544
unknown function (ip: 0x7f63d4694ac2) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7f63d472684f) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7f63d4691115) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/builder-amdci5-5/julialang/julia-master/src/gc-stock.c:3544
unknown function (ip: 0x7f63d4694ac2) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7f63d472684f) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7f63d4691115) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/builder-amdci5-5/julialang/julia-master/src/scheduler.c:520
poptask at ./task.jl:1163
wait at ./task.jl:1172
task_done_hook at ./task.jl:839
jfptr_task_done_hook_97036.1 at /opt/hostedtoolcache/julia/nightly/x64/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/builder-amdci5-5/julialang/julia-master/src/julia.h:2240 [inlined]
jl_finish_task at /cache/build/builder-amdci5-5/julialang/julia-master/src/task.c:338
start_task at /cache/build/builder-amdci5-5/julialang/julia-master/src/task.c:1274
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7f63d4691115) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/builder-amdci5-5/julialang/julia-master/src/scheduler.c:520
poptask at ./task.jl:1163
wait at ./task.jl:1172
task_done_hook at ./task.jl:839

[3917] signal 15: Terminated
in expression starting at /home/runner/work/MPI.jl/MPI.jl/test/test_threads.jl:39
unknown function (ip: 0x7fb132e91115) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/builder-amdci5-5/julialang/julia-master/src/gc-stock.c:3544
unknown function (ip: 0x7fb132e94ac2) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7fb132f2684f) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7fb132e91115) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/builder-amdci5-5/julialang/julia-master/src/gc-stock.c:3544
unknown function (ip: 0x7fb132e94ac2) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7fb132f2684f) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7fb132e91115) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/builder-amdci5-5/julialang/julia-master/src/gc-stock.c:3544
unknown function (ip: 0x7fb132e94ac2) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7fb132f2684f) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7fb132e91115) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/builder-amdci5-5/julialang/julia-master/src/scheduler.c:520
poptask at ./task.jl:1163
wait at ./task.jl:1172
task_done_hook at ./task.jl:839
jfptr_task_done_hook_97036.1 at /opt/hostedtoolcache/julia/nightly/x64/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/builder-amdci5-5/julialang/julia-master/src/julia.h:2240 [inlined]
jl_finish_task at /cache/build/builder-amdci5-5/julialang/julia-master/src/task.c:338
start_task at /cache/build/builder-amdci5-5/julialang/julia-master/src/task.c:1274
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7fb132e91115) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/builder-amdci5-5/julialang/julia-master/src/scheduler.c:520
poptask at ./task.jl:1163
wait at ./task.jl:1172
task_done_hook at ./task.jl:839
jfptr_task_done_hook_97036.1 at /opt/hostedtoolcache/julia/nightly/x64/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/builder-amdci5-5/julialang/julia-master/src/julia.h:2240 [inlined]
jl_finish_task at /cache/build/builder-amdci5-5/julialang/julia-master/src/task.c:338
start_task at /cache/build/builder-amdci5-5/julialang/julia-master/src/task.c:1274
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7fb132e91115) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/builder-amdci5-5/julialang/julia-master/src/scheduler.c:520
poptask at ./task.jl:1163
wait at ./task.jl:1172
task_done_hook at ./task.jl:839
jfptr_task_done_hook_97036.1 at /opt/hostedtoolcache/julia/nightly/x64/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/builder-amdci5-5/julialang/julia-master/src/julia.h:2240 [inlined]
jl_finish_task at /cache/build/builder-amdci5-5/julialang/julia-master/src/task.c:338
start_task at /cache/build/builder-amdci5-5/julialang/julia-master/src/task.c:1274
unknown function (ip: (nil)) at (unknown file)
clock_nanosleep at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
__nanosleep at /lib/x86_64-linux-gnu/libc.so.6 (unknown line
[3916] signal 15: Terminated
in expression starting at /home/runner/work/MPI.jl/MPI.jl/test/test_threads.jl:18
unknown function (ip: 0x7fa2b4c91115) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/builder-amdci5-5/julialang/julia-master/src/gc-stock.c:3544
unknown function (ip: 0x7fa2b4c94ac2) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7fa2b4d2684f) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7fa2b4c91115) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/builder-amdci5-5/julialang/julia-master/src/gc-stock.c:3544
unknown function (ip: 0x7fa2b4c94ac2) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7fa2b4d2684f) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7fa2b4c91115) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/builder-amdci5-5/julialang/julia-master/src/gc-stock.c:3544
unknown function (ip: 0x7fa2b4c94ac2) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7fa2b4d2684f) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7fa2b4c91115) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/builder-amdci5-5/julialang/julia-master/src/scheduler.c:520
poptask at ./task.jl:1163
wait at ./task.jl:1172
task_done_hook at ./task.jl:839
jfptr_task_done_hook_97036.1 at /opt/hostedtoolcache/julia/nightly/x64/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/builder-amdci5-5/julialang/julia-master/src/julia.h:2240 [inlined]
jl_finish_task at /cache/build/builder-amdci5-5/julialang/julia-master/src/task.c:338
start_task at /cache/build/builder-amdci5-5/julialang/julia-master/src/task.c:1274
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7fa2b4c91115) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/builder-amdci5-5/julialang/julia-master/src/scheduler.c:520
poptask at ./task.jl:1163
wait at ./task.jl:1172
task_done_hook at ./task.jl:839
)
usleep at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
ompi_mpi_finalize at /home/runner/.julia/artifacts/c519b6f9838786c8d97506fb28f7e10dfc74b9a3/lib/libmpi.so (unknown line)
MPI_Finalize at /home/runner/work/MPI.jl/MPI.jl/src/api/generated_api.jl:1872 [inlined]
Finalize at /home/runner/work/MPI.jl/MPI.jl/src/environment.jl:263
unknown function (ip: 0x7fb0f9b1b51f) at (unknown file)
jl_apply at /cache/build/builder-amdci5-5/julialang/julia-master/src/julia.h:2240 [inlined]
do_call at /cache/build/builder-amdci5-5/julialang/julia-master/src/interpreter.c:125
eval_value at /cache/build/builder-amdci5-5/julialang/julia-master/src/interpreter.c:222
eval_stmt_value at /cache/build/builder-amdci5-5/julialang/julia-master/src/interpreter.c:173 [inlined]
eval_body at /cache/build/builder-amdci5-5/julialang/julia-master/src/interpreter.c:684
jl_interpret_toplevel_thunk at /cache/build/builder-amdci5-5/julialang/julia-master/src/interpreter.c:895
jl_toplevel_eval_flex at /cache/build/builder-amdci5-5/julialang/julia-master/src/toplevel.c:1065
jl_toplevel_eval_flex at /cache/build/builder-amdci5-5/julialang/julia-master/src/toplevel.c:1005
ijl_toplevel_eval at /cache/build/builder-amdci5-5/julialang/julia-master/src/toplevel.c:1076
ijl_toplevel_eval_in at /cache/build/builder-amdci5-5/julialang/julia-master/src/toplevel.c:1118
eval at ./boot.jl:460
include_string at ./loading.jl:2839
_include at ./loading.jl:2899

For the record, last successful run was with Julia Version 1.12.0-DEV.1263 Commit 17445fe752b (2024-09-29 09:41 UTC) (this may help with bisection). Version of OpenMPI_jll is v5.0.5+0 in both cases, so that doesn't seem to be relevant.

Originally posted by @giordano in #887 (comment)

Update: tests passed in https://github.com/JuliaParallel/MPI.jl/actions/runs/12728449076/job/35479046456?pr=887 with Julia Version 1.12.0-DEV.1872 Commit db8cc484fd8 (2025-01-10 20:34 UTC), so the issue may have been fixed already. Leaving the ticket open as a reminder to double check all is good.

[giordano]

I was under the impression the issue above was fixed at some point, but now we have a seemingly different error (abort instead of segfault) on Julia nightly, again with OpenMPI: https://github.com/JuliaParallel/MPI.jl/actions/runs/14725590941/job/41327425012?pr=901

realloc(): invalid old size

[4107] signal 6 (-6): Aborted
in expression starting at /home/runner/work/MPI.jl/MPI.jl/test/test_threads.jl:18
pthread_kill at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7fec3ec297b5) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7fec3eca8ff4) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7fec3ecad0ab) at /lib/x86_64-linux-gnu/libc.so.6
realloc at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_ZN4llvm15SmallVectorBaseIjE8grow_podEPvmm at /opt/hostedtoolcache/julia/nightly/x64/bin/../lib/julia/libjulia-internal.so.1.13 (unknown line)
grow_pod at /cache/build/builder-amdci5-2/julialang/julia-master/usr/include/llvm/ADT/SmallVector.h:152 [inlined]
grow at /cache/build/builder-amdci5-2/julialang/julia-master/usr/include/llvm/ADT/SmallVector.h:539 [inlined]
reserve at /cache/build/builder-amdci5-2/julialang/julia-master/usr/include/llvm/ADT/SmallVector.h:679 [inlined]
resizeImpl<false> at /cache/build/builder-amdci5-2/julialang/julia-master/usr/include/llvm/ADT/SmallVector.h:642 [inlined]
resizeImpl<false> at /cache/build/builder-amdci5-2/julialang/julia-master/usr/include/llvm/ADT/SmallVector.h:633 [inlined]
resize at /cache/build/builder-amdci5-2/julialang/julia-master/usr/include/llvm/ADT/SmallVector.h:652 [inlined]
allocLine at /cache/build/builder-amdci5-2/julialang/julia-master/src/coverage.cpp:34
jl_coverage_alloc_line at /cache/build/builder-amdci5-2/julialang/julia-master/src/coverage.cpp:53
operator() at /cache/build/builder-amdci5-2/julialang/julia-master/src/codegen.cpp:9076 [inlined]
_M_invoke at /usr/local/x86_64-linux-gnu/include/c++/9.1.0/bits/std_function.h:300
operator() at /usr/local/x86_64-linux-gnu/include/c++/9.1.0/bits/std_function.h:690 [inlined]
operator() at /cache/build/builder-amdci5-2/julialang/julia-master/src/codegen.cpp:9054 [inlined]
_M_invoke at /usr/local/x86_64-linux-gnu/include/c++/9.1.0/bits/std_function.h:300
operator() at /usr/local/x86_64-linux-gnu/include/c++/9.1.0/bits/std_function.h:690 [inlined]
operator() at /cache/build/builder-amdci5-2/julialang/julia-master/src/codegen.cpp:9054 [inlined]
_M_invoke at /usr/local/x86_64-linux-gnu/include/c++/9.1.0/bits/std_function.h:300 [inlined]
operator() at /usr/local/x86_64-linux-gnu/include/c++/9.1.0/bits/std_function.h:690 [inlined]
emit_function at /cache/build/builder-amdci5-2/julialang/julia-master/src/codegen.cpp:9080
jl_emit_code at /cache/build/builder-amdci5-2/julialang/julia-master/src/codegen.cpp:9709
jl_emit_codeinst at /cache/build/builder-amdci5-2/julialang/julia-master/src/codegen.cpp:9780
jl_emit_codeinst_to_jit_impl at /cache/build/builder-amdci5-2/julialang/julia-master/src/jitlayers.cpp:765
add_codeinsts_to_jit! at ./../usr/share/julia/Compiler/src/typeinfer.jl:1321
typeinf_ext_toplevel at ./../usr/share/julia/Compiler/src/typeinfer.jl:1328
typeinf_ext_toplevel at ./../usr/share/julia/Compiler/src/typeinfer.jl:1335
jfptr_typeinf_ext_toplevel_82887.1 at /opt/hostedtoolcache/julia/nightly/x64/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/builder-amdci5-2/julialang/julia-master/src/julia.h:2346 [inlined]
jl_type_infer at /cache/build/builder-amdci5-2/julialang/julia-master/src/gf.c:444
jl_compile_method_internal at /cache/build/builder-amdci5-2/julialang/julia-master/src/gf.c:2981
_jl_invoke at /cache/build/builder-amdci5-2/julialang/julia-master/src/gf.c:3471 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-2/julialang/julia-master/src/gf.c:3679
macro expansion at /home/runner/work/MPI.jl/MPI.jl/test/test_threads.jl:25 [inlined]
#4 at ./threadingconstructs.jl:276
#2 at ./threadingconstructs.jl:243 [inlined]
#threading_run##0 at ./threadingconstructs.jl:177
unknown function (ip: 0x7fec0c71940f) at (unknown file)
jl_apply at /cache/build/builder-amdci5-2/julialang/julia-master/src/julia.h:2346 [inlined]
start_task at /cache/build/builder-amdci5-2/julialang/julia-master/src/task.c:1249
Allocations: 2361218 (Pool: 2361146; Big: 72); GC: 3

[4105] signal 15: Terminated
in expression starting at /home/runner/work/MPI.jl/MPI.jl/test/test_threads.jl:39
unknown function (ip: 0x7f8fcce98d6f) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/builder-amdci5-2/julialang/julia-master/src/gc-stock.c:3624
unknown function (ip: 0x7f8fcce9caa3) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7f8fccf29c3b) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7f8fcce98d6f) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/builder-amdci5-2/julialang/julia-master/src/gc-stock.c:3624
unknown function (ip: 0x7f8fcce9caa3) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7f8fccf29c3b) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7f8fcce98d6f) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/builder-amdci5-2/julialang/julia-master/src/gc-stock.c:3624
unknown function (ip: 0x7f8fcce9caa3) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7f8fccf29c3b) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7f8fcce98d6f) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/builder-amdci5-2/julialang/julia-master/src/scheduler.c:520
wait at ./task.jl:1216
wait_forever at ./task.jl:1150
jfptr_wait_forever_30137.1 at /opt/hostedtoolcache/julia/nightly/x64/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/builder-amdci5-2/julialang/julia-master/src/julia.h:2346 [inlined]
start_task at /cache/build/builder-amdci5-2/julialang/julia-master/src/task.c:1249
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7f8fcce98d6f) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/builder-amdci5-2/julialang/julia-master/src/scheduler.c:520
wait at ./task.jl:1216
wait_forever at ./task.jl:1150

[4106] signal 15: Terminated
in expression starting at /home/runner/work/MPI.jl/MPI.jl/test/test_threads.jl:39
unknown function (ip: 0x7f9703498d6f) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/builder-amdci5-2/julialang/julia-master/src/gc-stock.c:3624
unknown function (ip: 0x7f970349caa3) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7f9703529c3b) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7f9703498d6f) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/builder-amdci5-2/julialang/julia-master/src/gc-stock.c:3624
unknown function (ip: 0x7f970349caa3) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7f9703529c3b) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7f9703498d6f) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/builder-amdci5-2/julialang/julia-master/src/gc-stock.c:3624
unknown function (ip: 0x7f970349caa3) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7f9703529c3b) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7f9703498d6f) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/builder-amdci5-2/julialang/julia-master/src/scheduler.c:520
wait at ./task.jl:1216
wait_forever at ./task.jl:1150
jfptr_wait_forever_30137.1 at /opt/hostedtoolcache/julia/nightly/x64/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/builder-amdci5-2/julialang/julia-master/src/julia.h:2346 [inlined]
start_task at /cache/build/builder-amdci5-2/julialang/julia-master/src/task.c:1249
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7f9703498d6f) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/builder-amdci5-2/julialang/julia-master/src/scheduler.c:520
wait at ./task.jl:1216
wait_forever at ./task.jl:1150
jfptr_wait_forever_30137.1 at /opt/hostedtoolcache/julia/nightly/x64/lib/julia/sys.so (unknown line)

[4104] signal 15: Terminated
in expression starting at /home/runner/work/MPI.jl/MPI.jl/test/test_threads.jl:18
unknown function (ip: 0x7fb8a9a98d6f) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/builder-amdci5-2/julialang/julia-master/src/gc-stock.c:3624
unknown function (ip: 0x7fb8a9a9caa3) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7fb8a9b29c3b) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7fb8a9a98d6f) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/builder-amdci5-2/julialang/julia-master/src/gc-stock.c:3624
unknown function (ip: 0x7fb8a9a9caa3) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7fb8a9b29c3b) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7fb8a9a98d6f) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/builder-amdci5-2/julialang/julia-master/src/gc-stock.c:3624
unknown function (ip: 0x7fb8a9a9caa3) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7fb8a9b29c3b) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7fb8a9a98d6f) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/builder-amdci5-2/julialang/julia-master/src/scheduler.c:520
wait at ./task.jl:1216
wait_forever at ./task.jl:1150
jfptr_wait_forever_30137.1 at /opt/hostedtoolcache/julia/nightly/x64/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/builder-amdci5-2/julialang/julia-master/src/julia.h:2346 [inlined]
start_task at /cache/build/builder-amdci5-2/julialang/julia-master/src/task.c:1249
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7fb8a9a98d6f) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/builder-amdci5-2/julialang/julia-master/src/scheduler.c:520
wait at ./task.jl:1216
wait_forever at ./task.jl:1150
jfptr_wait_forever_30137.1 at /opt/hostedtoolcache/julia/nightly/x64/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/builder-amdci5-2/julialang/julia-master/src/julia.h:2346 [inlined]
start_task at /cache/build/builder-amdci5-2/julialang/julia-master/src/task.c:1249
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7fb8a9a98d6f) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/builder-amdci5-2/julialang/julia-master/src/scheduler.c:520
wait at ./task.jl:1216
wait_forever at ./task.jl:1150
jfptr_wait_forever_30137.1 at /opt/hostedtoolcache/julia/nightly/x64/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/builder-amdci5-2/julialang/julia-master/src/julia.h:2346 [inlined]
start_task at /cache/build/builder-amdci5-2/julialang/julia-master/src/task.c:1249
unknown function (ip: (nil)) at (unknown file)
unknown function (ip: 0x7fb8a9a98d6f) at /lib/x86_64-linux-gnu/libc.so.6
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/builder-amdci5-2/julialang/julia-master/src/scheduler.c:520
--------------------------------------------------------------------------
prterun noticed that process rank 3 with PID 4107 on node fv-az1068-579 exited on
signal 6 (Aborted).
--------------------------------------------------------------------------

This is with

Julia Version 1.13.0-DEV.461
Commit dc8bc7366c5 (2025-04-28 23:29 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 4 × AMD EPYC 7763 64-Core Processor
  WORD_SIZE: 64
  LLVM: libLLVM-19.1.7 (ORCJIT, znver3)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 4 virtual cores)
Environment:
  JULIA_MPI_TEST_BINARY = OpenMPI_jll
  JULIA_MPI_TEST_ABI = OpenMPI

and

  [fe0851c0] OpenMPI_jll v5.0.7+2

There's a segfault on i686 with default configuration (so with MPICH_jll), but if we're lucky the underlying issue is the same:

[3812] signal 11 (1): Segmentation fault
in expression starting at /home/runner/work/MPI.jl/MPI.jl/test/test_threads.jl:18

[3811] signal 11 (1): Segmentation fault
in expression starting at /home/runner/work/MPI.jl/MPI.jl/test/test_threads.jl:18
allocLine at /cache/build/builder-amdci4-1/julialang/julia-master/src/coverage.cpp:39
jl_coverage_alloc_line at /cache/build/builder-amdci4-1/julialang/julia-master/src/coverage.cpp:53
malloc_consolidate(): unaligned fastbin chunk detected

[3813] signal 6 (-6): Aborted
in expression starting at /home/runner/work/MPI.jl/MPI.jl/test/test_threads.jl:18
__kernel_vsyscall at linux-gate.so.1 (unknown line)
unknown function (ip: 0xec52d016) at /lib32/libc.so.6
gsignal at /lib32/libc.so.6 (unknown line)
abort at /lib32/libc.so.6 (unknown line)
unknown function (ip: 0xec4c4369) at /lib32/libc.so.6
unknown function (ip: 0xec53852c) at /lib32/libc.so.6
unknown function (ip: 0xec538f4e) at /lib32/libc.so.6
unknown function (ip: 0xec53b586) at /lib32/libc.so.6
unknown function (ip: 0xec53c0ed) at /lib32/libc.so.6
realloc at /lib32/libc.so.6 (unknown line)
_ZN4llvm15SmallVectorBaseIjE8grow_podEPvjj at /opt/hostedtoolcache/julia/nightly/x86/bin/../lib/julia/libLLVM.so.19.1jl (unknown line)
_ZN4llvm13PMDataManager3addEPNS_4PassEb at /opt/hostedtoolcache/julia/nightly/x86/bin/../lib/julia/libLLVM.so.19.1jl (unknown line)
_ZN4llvm17PMTopLevelManager12schedulePassEPNS_4PassE at /opt/hostedtoolcache/julia/nightly/x86/bin/../lib/julia/libLLVM.so.19.1jl (unknown line)
_ZN4llvm17PMTopLevelManager12schedulePassEPNS_4PassE at /opt/hostedtoolcache/julia/nightly/x86/bin/../lib/julia/libLLVM.so.19.1jl (unknown line)
_ZN4llvm17PMTopLevelManager12schedulePassEPNS_4PassE at /opt/hostedtoolcache/julia/nightly/x86/bin/../lib/julia/libLLVM.so.19.1jl (unknown line)
_ZN4llvm16TargetPassConfig7addPassEPNS_4PassE at /opt/hostedtoolcache/julia/nightly/x86/bin/../lib/julia/libLLVM.so.19.1jl (unknown line)
operator() at /cache/build/builder-amdci4-1/julialang/julia-master/src/codegen.cpp:9076 [inlined]
_M_invoke at /usr/local/i686-linux-gnu/include/c++/9.1.0/bits/std_function.h:300
operator() at /usr/local/i686-linux-gnu/include/c++/9.1.0/bits/std_function.h:690 [inlined]
operator() at /cache/build/builder-amdci4-1/julialang/julia-master/src/codegen.cpp:9054 [inlined]
_M_invoke at /usr/local/i686-linux-gnu/include/c++/9.1.0/bits/std_function.h:300
operator() at /usr/local/i686-linux-gnu/include/c++/9.1.0/bits/std_function.h:690 [inlined]
operator() at /cache/build/builder-amdci4-1/julialang/julia-master/src/codegen.cpp:9054 [inlined]
_M_invoke at /usr/local/i686-linux-gnu/include/c++/9.1.0/bits/std_function.h:300 [inlined]
operator() at /usr/local/i686-linux-gnu/include/c++/9.1.0/bits/std_function.h:690 [inlined]
emit_function at /cache/build/builder-amdci4-1/julialang/julia-master/src/codegen.cpp:9080
jl_emit_code at /cache/build/builder-amdci4-1/julialang/julia-master/src/codegen.cpp:9709
jl_emit_codeinst at /cache/build/builder-amdci4-1/julialang/julia-master/src/codegen.cpp:9780
malloc_consolidate(): unaligned fastbin chunk detected

[3811] signal 6 (-6): Aborted
in expression starting at /home/runner/work/MPI.jl/MPI.jl/test/test_threads.jl:18
_ZN12_GLOBAL__N_113X86PassConfig31addRegAssignAndRewriteOptimizedEv at /opt/hostedtoolcache/julia/nightly/x86/bin/../lib/julia/libLLVM.so.19.1jl (unknown line)
_ZN4llvm16TargetPassConfig20addOptimizedRegAllocEv at /opt/hostedtoolcache/julia/nightly/x86/bin/../lib/julia/libLLVM.so.19.1jl (unknown line)
_ZN4llvm16TargetPassConfig16addMachinePassesEv at /opt/hostedtoolcache/julia/nightly/x86/bin/../lib/julia/libLLVM.so.19.1jl (unknown line)
jl_emit_codeinst_to_jit_impl at /cache/build/builder-amdci4-1/julialang/julia-master/src/jitlayers.cpp:765
__kernel_vsyscall at linux-gate.so.1 (unknown line)
unknown function (ip: 0xf4021016) at /lib32/libc.so.6
gsignal at /lib32/libc.so.6 (unknown line)
abort at /lib32/libc.so.6 (unknown line)
unknown function (ip: 0xf3fb8369) at /lib32/libc.so.6
unknown function (ip: 0xf402c52c) at /lib32/libc.so.6
unknown function (ip: 0xf402cf4e) at /lib32/libc.so.6
unknown function (ip: 0xf402f586) at /lib32/libc.so.6
unknown function (ip: 0xf40300ed) at /lib32/libc.so.6
realloc at /lib32/libc.so.6 (unknown line)
_ZN4llvm15SmallVectorBaseIjE8grow_podEPvjj at /opt/hostedtoolcache/julia/nightly/x86/bin/../lib/julia/libLLVM.so.19.1jl (unknown line)
_ZN4llvm13PMDataManager3addEPNS_4PassEb at /opt/hostedtoolcache/julia/nightly/x86/bin/../lib/julia/libLLVM.so.19.1jl (unknown line)
_ZN4llvm17PMTopLevelManager12schedulePassEPNS_4PassE at /opt/hostedtoolcache/julia/nightly/x86/bin/../lib/julia/libLLVM.so.19.1jl (unknown line)
_ZN4llvm16TargetPassConfig7addPassEPNS_4PassE at /opt/hostedtoolcache/julia/nightly/x86/bin/../lib/julia/libLLVM.so.19.1jl (unknown line)
_ZN4llvm16TargetPassConfig7addPassEPKv at /opt/hostedtoolcache/julia/nightly/x86/bin/../lib/julia/libLLVM.so.19.1jl (unknown line)
jl_add_codeinst_to_jit at /cache/build/builder-amdci4-1/julialang/julia-master/src/gf.c:2843
_ZN4llvm16TargetPassConfig16addMachinePassesEv at /opt/hostedtoolcache/julia/nightly/x86/bin/../lib/julia/libLLVM.so.19.1jl (unknown line)
_ZL23addPassesToGenerateCodeRN4llvm17LLVMTargetMachineERNS_6legacy15PassManagerBaseEbRNS_28MachineModuleInfoWrapperPassE at /opt/hostedtoolcache/julia/nightly/x86/bin/../lib/julia/libLLVM.so.19.1jl (unknown line)
_ZN4llvm17LLVMTargetMachine17addPassesToEmitMCERNS_6legacy15PassManagerBaseERPNS_9MCContextERNS_17raw_pwrite_streamEb at /opt/hostedtoolcache/julia/nightly/x86/bin/../lib/julia/libLLVM.so.19.1jl (unknown line)
_ZN4llvm3orc14SimpleCompilerclERNS_6ModuleE at /opt/hostedtoolcache/julia/nightly/x86/bin/../lib/julia/libLLVM.so.19.1jl (unknown line)
operator() at /cache/build/builder-amdci4-1/julialang/julia-master/src/jitlayers.cpp:1556
addModule at /cache/build/builder-amdci4-1/julialang/julia-master/src/jitlayers.cpp:2032
jl_compile_codeinst_now at /cache/build/builder-amdci4-1/julialang/julia-master/src/jitlayers.cpp:611
jl_compile_codeinst_impl at /cache/build/builder-amdci4-1/julialang/julia-master/src/jitlayers.cpp:819
jl_compile_method_internal at /cache/build/builder-amdci4-1/julialang/julia-master/src/gf.c:2994
_jl_invoke at /cache/build/builder-amdci4-1/julialang/julia-master/src/gf.c:3471 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-1/julialang/julia-master/src/gf.c:3679
macro expansion at /home/runner/work/MPI.jl/MPI.jl/test/test_threads.jl:26 [inlined]
#4 at ./threadingconstructs.jl:276
unknown function (ip: 0xc8f3cb4f) at (unknown file)
unknown function (ip: 0xc8f3c4f8) at (unknown file)
unknown function (ip: 0xc8f3c4c9) at (unknown file)
#2 at ./threadingconstructs.jl:243 [inlined]
#threading_run##0 at ./threadingconstructs.jl:177
unknown function (ip: 0xc8f3c3ee) at (unknown file)
_jl_invoke at /cache/build/builder-amdci4-1/julialang/julia-master/src/gf.c:3479 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-1/julialang/julia-master/src/gf.c:3679
jl_apply at /cache/build/builder-amdci4-1/julialang/julia-master/src/julia.h:2348 [inlined]
start_task at /cache/build/builder-amdci4-1/julialang/julia-master/src/task.c:1249
Allocations: 2629311 (Pool: 2629262; Big: 49); GC: 10
add_codeinsts_to_jit! at ./../usr/share/julia/Compiler/src/typeinfer.jl:1321
_ZL23addPassesToGenerateCodeRN4llvm17LLVMTargetMachineERNS_6legacy15PassManagerBaseEbRNS_28MachineModuleInfoWrapperPassE at /opt/hostedtoolcache/julia/nightly/x86/bin/../lib/julia/libLLVM.so.19.1jl (unknown line)
_ZN4llvm17LLVMTargetMachine17addPassesToEmitMCERNS_6legacy15PassManagerBaseERPNS_9MCContextERNS_17raw_pwrite_streamEb at /opt/hostedtoolcache/julia/nightly/x86/bin/../lib/julia/libLLVM.so.19.1jl (unknown line)
_ZN4llvm3orc14SimpleCompilerclERNS_6ModuleE at /opt/hostedtoolcache/julia/nightly/x86/bin/../lib/julia/libLLVM.so.19.1jl (unknown line)
operator() at /cache/build/builder-amdci4-1/julialang/julia-master/src/jitlayers.cpp:1556
addModule at /cache/build/builder-amdci4-1/julialang/julia-master/src/jitlayers.cpp:2032
jl_compile_codeinst_now at /cache/build/builder-amdci4-1/julialang/julia-master/src/jitlayers.cpp:630
jl_compile_codeinst_impl at /cache/build/builder-amdci4-1/julialang/julia-master/src/jitlayers.cpp:819
jl_compile_method_internal at /cache/build/builder-amdci4-1/julialang/julia-master/src/gf.c:2994
_jl_invoke at /cache/build/builder-amdci4-1/julialang/julia-master/src/gf.c:3471 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-1/julialang/julia-master/src/gf.c:3679
macro expansion at /home/runner/work/MPI.jl/MPI.jl/test/test_threads.jl:25 [inlined]
#4 at ./threadingconstructs.jl:276
#2 at ./threadingconstructs.jl:243 [inlined]
#threading_run##0 at ./threadingconstructs.jl:177
unknown function (ip: 0xd09873ee) at (unknown file)
_jl_invoke at /cache/build/builder-amdci4-1/julialang/julia-master/src/gf.c:3479 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-1/julialang/julia-master/src/gf.c:3679
jl_apply at /cache/build/builder-amdci4-1/julialang/julia-master/src/julia.h:2348 [inlined]
start_task at /cache/build/builder-amdci4-1/julialang/julia-master/src/task.c:1249
Allocations: 2632977 (Pool: 2632928; Big: 49); GC: 10

Version:

Julia Version 1.13.0-DEV.442
Commit a47b58f2668 (2025-04-24 15:22 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (i686-linux-gnu)
  CPU: 4 × AMD EPYC 7763 64-Core Processor
  WORD_SIZE: 32
  LLVM: libLLVM-19.1.7 (ORCJIT, znver3)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 4 virtual cores)