`--threads` option seems to now result in segfault in `jl_init_thread_heap` (`julia/src/gc.c:3992`)

[LebedevRI]

Building v1.11.0-beta1 (+ #54538) on debian sid, using clang-17:

$ cat /tmp/julia-multistage/build/stage-2/julia/Make.user 
LLVM_VERSION=17
USE_BINARYBUILDER_LLVM=1
EXTRA_CFLAGS=
OPENBLAS_TARGET_ARCH=ZEN
USE_BINARYBUILDER=1
CMAKE_GENERATOR=Ninja
override CC=clang-$(LLVM_VERSION)
override CXX=clang++-$(LLVM_VERSION)
MARCH=native
CFLAGS=-march=$(MARCH) -Wl,--undefined-version $(EXTRA_CFLAGS)
CXXFLAGS=$(CFLAGS)
JULIA_CPU_TARGET=$(MARCH)

but the result is

$ JULIA_NUM_THREADS=32 gdb /tmp/julia-multistage/build/stage-2/julia/usr/bin/julia
GNU gdb (Debian 13.2-1+b1) 13.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
No symbol table is loaded.  Use the "file" command.
Breakpoint 1 (__ubsan_handle_load_invalid_value) pending.
No symbol table is loaded.  Use the "file" command.
Breakpoint 2 (__asan_report_error) pending.
No symbol table is loaded.  Use the "file" command.
Breakpoint 3 (__ubsan::ScopedReport::~ScopedReport) pending.
No symbol table is loaded.  Use the "file" command.
Breakpoint 4 (__ubsan::Diag::~Diag) pending.
No symbol table is loaded.  Use the "file" command.
Breakpoint 5 (__sanitizer_print_stack_trace) pending.
No symbol table is loaded.  Use the "file" command.
Breakpoint 6 (^abort) pending.
No symbol table is loaded.  Use the "file" command.
Breakpoint 7 (SignalHandler) pending.
No symbol table is loaded.  Use the "file" command.
Breakpoint 8 (llvm::sys::PrintStackTrace) pending.
No symbol table is loaded.  Use the "file" command.
Breakpoint 9 (SignalHandler) pending.
No symbol table is loaded.  Use the "file" command.
Breakpoint 10 (^raise) pending.
No symbol table is loaded.  Use the "file" command.
Breakpoint 11 (llvm::sys::RunSignalHandlers) pending.
No symbol table is loaded.  Use the "file" command.
Breakpoint 12 (^__assert_fail) pending.
Reading symbols from /tmp/julia-multistage/build/stage-2/julia/usr/bin/julia...
(gdb) r --cpu-target native --optimize=3 --min-optlevel=3 --inline=yes -g 0 --threads 1
Starting program: /tmp/julia-multistage/build/stage-2/julia/usr/bin/julia --cpu-target native --optimize=3 --min-optlevel=3 --inline=yes -g 0 --threads 1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Detaching after fork from child process 424652]
[New Thread 0x7ffff00006c0 (LWP 424654)]
[New Thread 0x7fffddc006c0 (LWP 424655)]
[New Thread 0x7fffdd2006c0 (LWP 424656)]
[New Thread 0x7fffd48006c0 (LWP 424657)]
[New Thread 0x7fffcbe006c0 (LWP 424658)]
[New Thread 0x7fffc34006c0 (LWP 424659)]
[New Thread 0x7fffbaa006c0 (LWP 424660)]
[New Thread 0x7fffb20006c0 (LWP 424661)]
[New Thread 0x7fffa96006c0 (LWP 424662)]
[New Thread 0x7fffa0c006c0 (LWP 424663)]
[New Thread 0x7fff982006c0 (LWP 424664)]
[New Thread 0x7fff8f8006c0 (LWP 424665)]
[New Thread 0x7fff86e006c0 (LWP 424666)]
[New Thread 0x7fff7e4006c0 (LWP 424667)]
[New Thread 0x7fff75a006c0 (LWP 424668)]
[New Thread 0x7fff6d0006c0 (LWP 424669)]
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-beta1.1 (2024-05-22)
 _/ |\__'_|_|_|\__'_|  |  v1.11.0-beta1-fix/d2bf60efdf* (fork: 117 commits, 96 days)
|__/                   |

julia> exit()
[Thread 0x7fff6d0006c0 (LWP 424669) exited]
[Thread 0x7fff75a006c0 (LWP 424668) exited]
[Thread 0x7fff7e4006c0 (LWP 424667) exited]
[Thread 0x7fff86e006c0 (LWP 424666) exited]
[Thread 0x7fff8f8006c0 (LWP 424665) exited]
[Thread 0x7fff982006c0 (LWP 424664) exited]
[Thread 0x7fffa0c006c0 (LWP 424663) exited]
[Thread 0x7fffa96006c0 (LWP 424662) exited]
[Thread 0x7fffb20006c0 (LWP 424661) exited]
[Thread 0x7fffbaa006c0 (LWP 424660) exited]
[Thread 0x7fffc34006c0 (LWP 424659) exited]
[Thread 0x7fffcbe006c0 (LWP 424658) exited]
[Thread 0x7fffd48006c0 (LWP 424657) exited]
[Thread 0x7fffdd2006c0 (LWP 424656) exited]
[Thread 0x7fffddc006c0 (LWP 424655) exited]
[Thread 0x7ffff00006c0 (LWP 424654) exited]
[Inferior 1 (process 424649) exited normally]
(gdb) r --cpu-target native --optimize=3 --min-optlevel=3 --inline=yes -g 0 --threads 2
Starting program: /tmp/julia-multistage/build/stage-2/julia/usr/bin/julia --cpu-target native --optimize=3 --min-optlevel=3 --inline=yes -g 0 --threads 2
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Detaching after fork from child process 424671]
[New Thread 0x7ffff00006c0 (LWP 424673)]
[New Thread 0x7fffe16006c0 (LWP 424674)]

Thread 3 "julia" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe16006c0 (LWP 424674)]
0x00007ffff6fb95c3 in jl_init_thread_heap (ptls=ptls@entry=0x7fffdc000b70) at /repositories/julia/src/gc.c:3992
3992        memset(&ptls->gc_num, 0, sizeof(ptls->gc_num));
(gdb) bt full
#0  0x00007ffff6fb95c3 in jl_init_thread_heap (ptls=ptls@entry=0x7fffdc000b70) at /repositories/julia/src/gc.c:3992
        p = 0x7fffdc000e48
        heap = <optimized out>
        mq = 0x7fffdc0019b0
        cq = 0x7fffdc0019b0
        gc_cache = 0x7fffdc001c30
        wsa = <optimized out>
        q = 0x7fffdc001a70
        wsa2 = 0x7fffdc003e20
#1  0x00007ffff6fa2fb2 in jl_init_threadtls (tid=1) at /repositories/julia/src/threading.c:382
        ptls = 0x7fffdc000b70
        bt_data = <optimized out>
        allstates = <optimized out>
#2  0x00007ffff6fa4238 in jl_threadfun (arg=0x5555556ddd70) at /repositories/julia/src/scheduler.c:181
        stack_lo = 0x0
        stack_hi = 0x0
        targ = 0x5555556ddd70
        ptls = <optimized out>
        ct = <optimized out>
        wasrunning = <optimized out>
#3  0x00007ffff7e37dbb in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
        ret = <optimized out>
        pd = <optimized out>
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737352268544, 969745306413698405, -128, 0, 140737488342352, 140736966164480, -969689506164503195, -969727538008421019}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, 
              canceltype = 0}}}
        not_first_call = <optimized out>
#4  0x00007ffff7eb99f8 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
No locals.
(gdb) 

[LebedevRI]

I've just tried with v1.10.3, and it does not appear to happen there.

(I'm also may be seeing some weird behavior around precompiled package when passing --optimize=3,
but i'm not sure.)
Also, apparently build directory is (no longer?) relocatable?

[JeffBezanson]

This looks unusual; I can't reproduce it. Maybe a build inconsistency? Or maybe clang-specific? Does it work with gcc?

[LebedevRI]

Thanks for taking a look!
I don't know what a "build inconsistency" means in this context,
but i'm not sure that is it, i start from a clean build directory.
Let me see if it works with gcc, but that would likely mean there's UB.

[LebedevRI]

@JeffBezanson yup :( Does not seem to immediately crash with gcc. This is rather quite worrying.

[LebedevRI]

Still broken in 1.11.1
It's quite concerning that such kind of issue has not been triaged and slipped into a production release.
Let me bisect it.

[LebedevRI]

$ git bisect log
git bisect start
# status: waiting for both good and bad commits
# bad: [8f5b7ca12ad48c6d740e058312fc8cf2bbe67848] set VERSION to 1.11.1 (#56180)
git bisect bad 8f5b7ca12ad48c6d740e058312fc8cf2bbe67848
# status: waiting for good commit(s), bad commit known
# good: [6f3fdf7b36250fb95f512a2b927ad2518c07d2b5] set VERSION to 1.10.5 (#55526)
git bisect good 6f3fdf7b36250fb95f512a2b927ad2518c07d2b5
# good: [0ba6ec2d2282937a084d7e5e5a0b026dc953bb31] Restore link to list of packages in Base docs (#50353)
git bisect good 0ba6ec2d2282937a084d7e5e5a0b026dc953bb31
# good: [f96585b5c9232b0029b93374543f69b30f5757b3] reflection: refactor `@assume_effects` implementation (#52399)
git bisect good f96585b5c9232b0029b93374543f69b30f5757b3
# good: [3a283d95d45d585a8727a6c7be6dc7c2966fe0de] document Base.Chars arguments of startswith/endswith (#52741)
git bisect good 3a283d95d45d585a8727a6c7be6dc7c2966fe0de
# bad: [4caa877589a7ec9f1883c6b2cdff7f952ab3e72b] clamp: Turn `ifelse` into ternary (#54038)
git bisect bad 4caa877589a7ec9f1883c6b2cdff7f952ab3e72b
# bad: [12add8d61f5e1650ef5dc2e20779b1ba1f8680b4] typo fix in scoped values docs (#53629)
git bisect bad 12add8d61f5e1650ef5dc2e20779b1ba1f8680b4
# good: [1b25d1e1e461dbd4b13d73dcc9ec8b38741fe5c8] Add CartesianIndex to getindex docstring (#51567)
git bisect good 1b25d1e1e461dbd4b13d73dcc9ec8b38741fe5c8
# good: [79421de930a470f67a044ea6aa29edb9c233d430] Avoid compiler warning about redefining jl_globalref_t (#53499)
git bisect good 79421de930a470f67a044ea6aa29edb9c233d430
# bad: [fb60a88cc9a633e1b17207b275520c11746c16f1] fix error path in `precompilepkgs` (#53606)
git bisect bad fb60a88cc9a633e1b17207b275520c11746c16f1
# good: [7859e32979b82571d584a1b02bac82826f4a0042] use more efficient `_readdirx` for tab completion (#53540)
git bisect good 7859e32979b82571d584a1b02bac82826f4a0042
# bad: [d30d8ef5d6af22b7a42eb510dcc73b1e66ea68bc] Update Documenter to v1.3 for inventory writing (#53571)
git bisect bad d30d8ef5d6af22b7a42eb510dcc73b1e66ea68bc
# good: [c01ef981d3843b50639e24bc3716ccb0fc18209d] Always return a value in 1-d circshift! of abstractarray.jl (#53554)
git bisect good c01ef981d3843b50639e24bc3716ccb0fc18209d
# bad: [53180e48ad973e5fa479327fb22ff2e1e5dba284] yet more atomics & cache-line fixes on work-stealing queue (#53424)
git bisect bad 53180e48ad973e5fa479327fb22ff2e1e5dba284
# first bad commit: [53180e48ad973e5fa479327fb22ff2e1e5dba284] yet more atomics & cache-line fixes on work-stealing queue (#53424)

So 53180e4 / #53424. CC @d-netto @vchuravy

But it does not appear to reproduce in master. Faulty cherry-pick or was it fixed later on? I did not bisect that.
Can that cherry-pick please be reverted? @vchuravy

Still happens on:

commit 6be0afc3f76955aef3227c7dab8edd4c2ac8e065 (HEAD, upstream/backports-release-1.11)
Author: Ian Butterworth <i.r.butterworth@gmail.com>
Date:   Wed Oct 23 19:14:02 2024 -0400

    REPL: don't complete str and cmd macros when the input matches the internal name like `r_` to `r"` (#56254)
    
    (cherry picked from commit 4236a33bc5a33dd123a8ffaf2ed2b4fe5641bb87)

and does not happen with

commit a2a782543060713c3a96024a082da26ca744aae3 (HEAD)
Author: Roman Lebedev <lebedev.ri@gmail.com>
Date:   Sat Oct 26 02:06:09 2024 +0300

    Revert "yet more atomics & cache-line fixes on work-stealing queue (#53424)"
    
    This reverts commit 53180e48ad973e5fa479327fb22ff2e1e5dba284.

commit 6be0afc3f76955aef3227c7dab8edd4c2ac8e065 (upstream/backports-release-1.11)
Author: Ian Butterworth <i.r.butterworth@gmail.com>
Date:   Wed Oct 23 19:14:02 2024 -0400

    REPL: don't complete str and cmd macros when the input matches the internal name like `r_` to `r"` (#56254)
    
    (cherry picked from commit 4236a33bc5a33dd123a8ffaf2ed2b4fe5641bb87)

[d-netto]

We should understand why this is not reproducible on master and what was the "fix".

Would help to get some data on what was the "fix" commit.

I can try help bisecting for the "fix" if you provide a versioninfo besides your Make.user.

[LebedevRI]

We should understand why this is not reproducible on master and what was the "fix".

Ack.

Would help to get some data on what was the "fix" commit.

I can try help bisecting for the "fix" if you provide a versioninfo besides your Make.user.

I'm not sure what you mean by versioninfo? Currently trying to start bisecting this...

[LebedevRI]

$ git bisect log 
git bisect start
# status: waiting for both good and bad commits
# good: [4d5a20d39f9d142652c6c4ebbfbe1ed09ea1d066] Docs: Change "Packages that wrap external libraries or programs should be named after those libraries or programs" to "Packages that wrap external libraries or programs can be named after those libraries or programs" (#53567)
git bisect good 4d5a20d39f9d142652c6c4ebbfbe1ed09ea1d066
# status: waiting for bad commit, 1 good commit known
# bad: [973b4ff9abb6e12cada196a9c97a96b358275e16] yet more atomics & cache-line fixes on work-stealing queue (#53424)
git bisect bad 973b4ff9abb6e12cada196a9c97a96b358275e16
# first bad commit: [973b4ff9abb6e12cada196a9c97a96b358275e16] yet more atomics & cache-line fixes on work-stealing queue (#53424)

Now let's see what fixed it...

[LebedevRI]

$ git bisect log
git bisect start '--term-new=fixed' '--term-old=unfixed'
# status: waiting for both good and bad commits
# unfixed: [973b4ff9abb6e12cada196a9c97a96b358275e16] yet more atomics & cache-line fixes on work-stealing queue (#53424)
git bisect unfixed 973b4ff9abb6e12cada196a9c97a96b358275e16
# status: waiting for bad commit, 1 good commit known
# fixed: [f1a90e076186e77792c77e2e2f0905e14b19f321] 🤖 [master] Bump the Pkg stdlib from 799dc2d54 to 116ba910c (#56336)
git bisect fixed f1a90e076186e77792c77e2e2f0905e14b19f321
# unfixed: [ce4f090e1731c5efd1da391469d635c6919a3955] Make ScopedValues public (#54574)
git bisect unfixed ce4f090e1731c5efd1da391469d635c6919a3955
# fixed: [68d04bad49515ea6f246f826be0fec7445e3e7ba] codegen: make boundscheck GEP not be inbounds while the load GEP is inbounds (#55681)
git bisect fixed 68d04bad49515ea6f246f826be0fec7445e3e7ba
# fixed: [f38015f470a5fe323cfc71f711f11677dcc80599] codegen: NFC refactoring to use Align type
git bisect fixed f38015f470a5fe323cfc71f711f11677dcc80599
# fixed: [ef19235df8ef505e21d70e9d81fe5f03fe1c333f] 🤖 [master] Bump the Tar stdlib from 81888a3 to 1114260 (#54670)
git bisect fixed ef19235df8ef505e21d70e9d81fe5f03fe1c333f
# fixed: [b88f64f16c454c238c9fa0ae858ca02b7084f329] make codegen threadsafe, sinking the necessary lock now into JuliaOJIT (#55106)
git bisect fixed b88f64f16c454c238c9fa0ae858ca02b7084f329
# fixed: [ec9001214829f3094af60b8d59574993d1cc78a9] Add fast method for copyto!(::Memory, ::Memory) (#55082)
git bisect fixed ec9001214829f3094af60b8d59574993d1cc78a9
# unfixed: [0d9404fae250d6979c1fc86aebdea265aeb7cda1] Declare type for `libgcc_s` handles in CompilerSupportLibraries_jll (#55011)
git bisect unfixed 0d9404fae250d6979c1fc86aebdea265aeb7cda1
# fixed: [082e14246d714fa01bc7d0b287f340c8318c75f9] remove unused managed realloc (#55050)
git bisect fixed 082e14246d714fa01bc7d0b287f340c8318c75f9
# fixed: [e318166860a3a67bee7d930be9d360b4c72ddd46] cleanup remset logic a bit (#55021)
git bisect fixed e318166860a3a67bee7d930be9d360b4c72ddd46
# unfixed: [712231199b3f3775b7734065dbfe478e77467e8a] Some mailmap updates (#55048)
git bisect unfixed 712231199b3f3775b7734065dbfe478e77467e8a
# first fixed commit: [e318166860a3a67bee7d930be9d360b4c72ddd46] cleanup remset logic a bit (#55021)

So e318166 / #55021.
To me, that does not look like something that would be an obvious fix,
so i still insist that the revert should be merged, please :)

[d-netto]

I'm not sure what you mean by versioninfo?

help?> versioninfo
search: versioninfo varinfo

  versioninfo(io::IO=stdout; verbose::Bool=false)

  Print information about the version of Julia in use. The output is
  controlled with boolean keyword arguments:

    •  verbose: print all additional information

  │ Warning
  │
  │  The output of this function may contain sensitive information.
  │  Before sharing the output, please review the output and remove any
  │  data that should not be shared publicly.

  See also: VERSION.

[LebedevRI]

Right. I'm guessing you don't actually want the version (i.e. commit) info, but the rest of the data.

julia> versioninfo()
Julia Version 1.12.0-DEV.822
Commit 712231199b (2024-07-05 20:54 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: 32 × AMD Ryzen 9 5950X 16-Core Processor
  WORD_SIZE: 64
  LLVM: libLLVM-17.0.6 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 32 virtual cores)

More specifically:

$ JULIA_NUM_THREADS=32 /builddirs/julia/julia --cpu-target native --threads 1 --procs 32 
< ok >
$ JULIA_NUM_THREADS=32 /builddirs/julia/julia --cpu-target native --threads 32 --procs 32 
Segmentation fault

[d-netto]

Yes, was mostly interested on what architecture you were running.