linux32 segfault with assertions enabled

[staticfloat]

My SHA upgrade PR is segfaulting on linux32 when assertions are enabled. The minimal reproducer is:

deepcopy(ntuple((i) -> 0x61, 1000000))

which occurs within the new SHA test script. It also appears this only happens on builds with assertions turned on. For instance, if I download a recent master build for linux32, it doesn't break:

$ rm -rf julia-9564bfbd6c/
$ curl -L 'https://s3.amazonaws.com/julialangnightlies/pretesting/linux/x86/1.8/julia-9564bfbd6c-linux32.tar.gz' | tar zx
$ ./julia-9564bfbd6c/bin/julia -e 'deepcopy(ntuple((i) -> 0x61, 1000000))'
$

However, if I just change that to assert_pretesting, to get the build with LLVM_ASSERTIONS=1 FORCE_ASSERTIONS=1, I suddenly get segfaults:

$ rm -rf julia-9564bfbd6c/
$ curl -L 'https://s3.amazonaws.com/julialangnightlies/assert_pretesting/linux/x86/1.8/julia-9564bfbd6c-linux32.tar.gz' | tar zx
$ ./julia-9564bfbd6c/bin/julia -e 'deepcopy(ntuple((i) -> 0x61, 1000000))'

signal (11): Segmentation fault
in expression starting at none:1
verify_type at /buildworker/worker/package_linux32/build/src/gf.c:2214 [inlined]
_jl_invoke at /buildworker/worker/package_linux32/build/src/gf.c:2246 [inlined]
jl_apply_generic at /buildworker/worker/package_linux32/build/src/gf.c:2427
jl_apply at /buildworker/worker/package_linux32/build/src/julia.h:1787 [inlined]
do_call at /buildworker/worker/package_linux32/build/src/interpreter.c:125
eval_value at /buildworker/worker/package_linux32/build/src/interpreter.c:214
eval_stmt_value at /buildworker/worker/package_linux32/build/src/interpreter.c:165 [inlined]
eval_body at /buildworker/worker/package_linux32/build/src/interpreter.c:597
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux32/build/src/interpreter.c:727
jl_toplevel_eval_flex at /buildworker/worker/package_linux32/build/src/toplevel.c:885
jl_toplevel_eval_flex at /buildworker/worker/package_linux32/build/src/toplevel.c:830
jl_toplevel_eval at /buildworker/worker/package_linux32/build/src/toplevel.c:894
jl_toplevel_eval_in at /buildworker/worker/package_linux32/build/src/toplevel.c:944
eval at ./boot.jl:373 [inlined]
exec_options at ./client.jl:268
_start at ./client.jl:495
jfptr__start_37152.clone_1 at /tmp/broken_julia/julia-9564bfbd6c/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux32/build/src/gf.c:2226 [inlined]
jl_apply_generic at /buildworker/worker/package_linux32/build/src/gf.c:2427
jl_apply at /buildworker/worker/package_linux32/build/src/julia.h:1787 [inlined]
true_main at /buildworker/worker/package_linux32/build/src/jlapi.c:559
jl_repl_entrypoint at /buildworker/worker/package_linux32/build/src/jlapi.c:701
main at /buildworker/worker/package_linux32/build/cli/loader_exe.c:42
__libc_start_main at /lib32/libc.so.6 (unknown line)
_start at ./julia-9564bfbd6c/bin/julia (unknown line)
Allocations: 100373 (Pool: 100343; Big: 30); GC: 1
Segmentation fault (core dumped)

Note that 9564bfb is the commit that my PR is based on top of, so the issue exists within Base julia already, it's just that the SHA test suite changes in that PR, and whatever code path it took previously did not trigger this issue.

[staticfloat]

Unfortunately, I can't get an rr trace for this, because it's a 32-bit process. :(

[staticfloat]

Running this inside of gdb, it looks like verify_type has been inlined into jl_apply_generic, and we're crashing here:

   0xf7a42175 <+629>:   jne    0xf7a425a3 <jl_apply_generic+1699>
   0xf7a4217b <+635>:   mov    (%ecx),%eax
   0xf7a4217d <+637>:   mov    %esi,0xc(%esp)
   0xf7a42181 <+641>:   mov    0x10(%ebp),%ecx
   0xf7a42184 <+644>:   mov    %ecx,0x8(%esp)
   0xf7a42188 <+648>:   mov    0xc(%ebp),%edi
   0xf7a4218b <+651>:   mov    %edi,0x4(%esp)
   0xf7a4218f <+655>:   mov    0x8(%ebp),%ecx
   0xf7a42192 <+658>:   mov    %ecx,(%esp)
   0xf7a42195 <+661>:   call   *%eax
   0xf7a42197 <+663>:   test   %eax,%eax
   0xf7a42199 <+665>:   je     0xf7a42110 <jl_apply_generic+528>
   0xf7a4219f <+671>:   mov    -0x4(%eax),%edx
   0xf7a421a2 <+674>:   and    $0xfffffff0,%edx
   0xf7a421a5 <+677>:   je     0xf7a42110 <jl_apply_generic+528>
   0xf7a421ab <+683>:   mov    -0x4(%edx),%edx
   0xf7a421ae <+686>:   and    $0xfffffff0,%edx
   0xf7a421b1 <+689>:   mov    -0xd4(%ebx),%ecx
=> 0xf7a421b7 <+695>:   cmp    %edx,(%ecx)

$ecx is 0x61616161, which on first glance looks like some kind of debugging value, but on second thought, could somehow be due to reading out from the middle of the giant tuple, as it is full of 0x61's. 😂 Changing the values within the tuple confirms that we are indeed trying to dereference a pointer whose contents have somehow been taken from or were overwritten by the contents of our tuple.

The jl_apply() call itself is the actual deepcopy() call, as expected:

(gdb) f 3
#3  0xf7a6354d in jl_apply (nargs=<optimized out>, args=0xffffb528) at /buildworker/worker/package_linux32/build/src/julia.h:1787
1787    in /buildworker/worker/package_linux32/build/src/julia.h
(gdb) p jl_(args[0])
Base.deepcopy
$10 = void
(gdb) p jl_(args[1])
(0x61, 0x61, 0x61, 0x61, 0x61, 0x61, 0x61, 0x61, 0x61, 0x61, 0x61, 0x61, 0x61, 0x61, 0x61, 0x61, 0x61, ...)
$11 = void

Experimenting with the MWE, I have found there is a particular tuple size above which the crash is reproducible, and below that size, the issue does not occur:

$ ./julia-9564bfbd6c/bin/julia -e 'deepcopy(ntuple((i) -> 0x61, 32744))'
$
$ ./julia-9564bfbd6c/bin/julia -e 'deepcopy(ntuple((i) -> 0x61, 32745))'
julia: /buildworker/worker/package_linux32/build/src/gf.c:2214: verify_type: Assertion `v && ((jl_value_t*)(((jl_taggedvalue_t*)((char*)(v) - sizeof(jl_taggedvalue_t)))->header & ~(uintptr_t)15)) && ((jl_value_t*)(((jl_taggedvalue_t*)((char*)(((jl_value_t*)(((jl_taggedvalue_t*)((char*)(v) - sizeof(jl_taggedvalue_t)))->header & ~(uintptr_t)15))) - sizeof(jl_taggedvalue_t)))->header & ~(uintptr_t)15)) == (jl_value_t*)jl_datatype_type' failed.

signal (6): Aborted
in expression starting at none:1
__kernel_vsyscall at linux-gate.so.1 (unknown line)
gsignal at /lib32/libc.so.6 (unknown line)
abort at /lib32/libc.so.6 (unknown line)
unknown function (ip: 0xf7cd72ba)
__assert_fail at /lib32/libc.so.6 (unknown line)
verify_type at /buildworker/worker/package_linux32/build/src/gf.c:2214
...

Note that now, we get an abort (due to the failed assert()) instead of a segfault; we have to raise the tuple size a bit more to get a segfault:

$ ./julia-9564bfbd6c/bin/julia -e 'deepcopy(ntuple((i) -> 0x61, 32976))'
<abort>
$ ./julia-9564bfbd6c/bin/julia -e 'deepcopy(ntuple((i) -> 0x61, 32977))'
<segfault>

Suspiciously close to 32K; perhaps we're overrunning some internal buffer.

[staticfloat]

I managed to bisect this to c0f8aef, which is a little surprising since it doesn't look that suspicious to me. @vtjnash any ideas on why this would be happening on i686?

You can replicate this on a Debian 9 docker image with the following steps:

FROM i386/debian:9

RUN apt update -y && apt install -y build-essential libatomic1 python gfortran perl wget m4 cmake pkg-config curl git
WORKDIR /build
RUN git clone https://github.com/JuliaLang/julia .
RUN git checkout c0f8aef3bfcf59699ee0f4db46f183fcd8dde0c1
RUN make -j30 MARCH=pentium4 LLVM_ASSERTIONS=1 FORCE_ASSERTIONS=1 VERBOSE=1 JULIA_PRECOMPILE=0

$ docker run -ti i386_test:debian9 bash
root@c8b07f26516b:/build# ./julia -e 'deepcopy(ntuple((i) -> 0xaf, 1000000))'

signal (11): Segmentation fault
...

[vchuravy]

Unfortunately, I can't get an rr trace for this, because it's a 32-bit process. :(

You should still be able to get an rr trace, there is a build option for 32bit

[staticfloat]

Okay, so building rr within the container, I can now get an rr trace:

# Dockerfile
FROM i386/debian:9

RUN apt update -y && apt install -y build-essential libatomic1 python gfortran perl wget m4 cmake pkg-config curl git ninja-build capnproto libcapnp-dev python3-pexpect gdb
WORKDIR /build

# Build rr
RUN git clone https://github.com/rr-debugger/rr /build/rr
WORKDIR /build/rr/_build
RUN cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=OFF -DWILL_RUN_TESTS=OFF -Dstaticlibs=ON ..
RUN make -j30

# Build julia
RUN git clone https://github.com/JuliaLang/julia /build/julia
WORKDIR /build/julia
RUN git checkout c0f8aef3bfcf59699ee0f4db46f183fcd8dde0c1
RUN make -j30 MARCH=pentium4 LLVM_ASSERTIONS=1 FORCE_ASSERTIONS=1 VERBOSE=1 JULIA_PRECOMPILE=0

$ docker build -t i386_test:debian9 .
...
$ docker run --privileged -ti i386_test:debian9 bash
...
root@352d81dfdc8b:/build/julia# /build/rr/_build/bin/rr record ./julia -e 'deepcopy(ntuple((i) -> 0xaf, 1000000))'

If you do this, you can get a trace like I did here: https://julialang-dumps.s3.amazonaws.com/linux32/_manual/stacksize-trace.tar.zst