SegFault in inference0.ji compilation on master for ARM

[yuyichao]

I get a segfault very early in the bootstrap when compiling the current master for 32bit ARM with stock LLVM 3.6.2. Happens with both qemu and native compilation.

Backtrace is

#0  0xb62d45b8 in jl_f_new_expr (F=<optimized out>, args=0xbefff254, nargs=3)
    at /home/yuyichao/julia/src/alloc.c:879
#1  0xb6ff60e8 in julia.new_0 ()
#2  0xb62787b4 in jl_apply (nargs=2, args=0xbefff2cc, f=<optimized out>)
    at /home/yuyichao/julia/src/julia.h:1328
#3  jl_apply_generic (F=0x95339680, args=0xbefff2cc, nargs=2)
    at /home/yuyichao/julia/src/gf.c:1708
#4  0xb62d1260 in jl_apply (nargs=2, args=0xbefff2cc, f=0x95339680)
    at /home/yuyichao/julia/src/julia.h:1328
#5  do_call (f=0x95339680, args=args@entry=0x9533c974, nargs=nargs@entry=2, 
    eval0=eval0@entry=0x0, locals=locals@entry=0x0, nl=nl@entry=0, 
    ngensym=ngensym@entry=0) at /home/yuyichao/julia/src/interpreter.c:65
#6  0xb62d03c0 in eval (e=<optimized out>, locals=locals@entry=0x0, 
    nl=nl@entry=0, ngensym=ngensym@entry=0)
    at /home/yuyichao/julia/src/interpreter.c:213
#7  0xb62d1170 in jl_interpret_toplevel_expr (e=<optimized out>)
    at /home/yuyichao/julia/src/interpreter.c:27
#8  0xb62e6934 in jl_toplevel_eval_flex (e=<optimized out>, fast=fast@entry=1)
    at /home/yuyichao/julia/src/toplevel.c:527
#9  0xb62e7734 in jl_toplevel_eval_flex (fast=1, e=<optimized out>)
    at /home/yuyichao/julia/src/toplevel.c:573
#10 jl_parse_eval_all (fname=fname@entry=0xb6d63a50 "boot.jl", len=len@entry=8)
    at /home/yuyichao/julia/src/toplevel.c:577
#11 0xb62e7970 in jl_load (fname=0xb6d63a50 "boot.jl", len=len@entry=8)
    at /home/yuyichao/julia/src/toplevel.c:617
#12 0xb62d73c8 in _julia_init (rel=<optimized out>)
    at /home/yuyichao/julia/src/init.c:549
#13 0xb62d8284 in julia_init (rel=<optimized out>)
    at /home/yuyichao/julia/src/task.c:260
#14 0x00011188 in main (argc=1, argv=0xbefffc1c)
    at /home/yuyichao/julia/ui/repl.c:594

The direct cause of the SegFault is that the first parameter (args[0]) to jl_f_new_expr is NULL.
This happens in the call _new(:LabelNode, :Int) in boot.jl (which is the #3 in the backtrace)

The jl_f_new_expr is called from julia.new_0 and the llvm ir can be find here, the dump of jl_Module right after codegen in to_function() is also available

AFAICT, the first argument should be *'jl_sym#curly5' and I've verified in GDB that this is indeed 0. A list of global symbols recognized by gdb is here and AFAIK none of them looks right. The dereference is wrong here but it seems that this is exactly the mistake we are making in codegen.

(gdb) p jl_(*'jl_sym#::4')
:fastmath                                                                         
$4 = void                                                                         
(gdb) p jl_(*'jl_sym#=1')
:Base
$5 = void
(gdb) p jl_(*'jl_sym#Type6')
:parameters
$6 = void
(gdb) p jl_(*'jl_sym#block8')
:flags
$7 = void
(gdb) p jl_(*'jl_sym#call2')
:meta
$8 = void
(gdb) p jl_(*'jl_sym#curly5')
#<null>
$9 = void
(gdb) p jl_(*'jl_sym#n7')
#<null>
$10 = void
(gdb) p jl_(*'jl_sym#new12')
:item
$11 = void

This only happens on master and not release-0.4 so @vtjnash

Also @ViralBShah if you can reproduce this too.

[yuyichao]

This is also the only function generated before the segfault...

[yuyichao]

(gdb) p jl_(*'jl_global#3')
:.
$12 = void
(gdb) p jl_(*'jl_global#9')
:boot.jl
$13 = void
(gdb) p jl_(*'jl_method#10')

Program received signal SIGSEGV, Segmentation fault.
jl_static_show_x (depth=1, v=0xb6278600 <jl_apply_generic>, out=<optimized out>)
    at /home/yuyichao/julia/src/builtins.c:1419
1419        else if (jl_is_cpointer(v)) {

Other global also looks messed up.

[yuyichao]

Also ref #11878 (comment), #12082 (and maybe also https://youtu.be/oX0Hl5FL7IM for anyone who want to debug this further)

[yuyichao]

Hmm. I think I just confused myself about global variable in gdb. It turns out that the global variables actually has the correct values

(gdb) p jl_('jl_sym#curly5')
:curly

and what I was getting with *'jl_sym#curly5' is just jl_sym_t::left which explains why they are all jl_sym_t or NULL even though they are all wrong...

However, it seems that the codegen is making the same mistake ;-p. Printing the arguments and the dereference (i.e. jl_sym_t::left) of the corresponding global variable.

(gdb) p jl_(args[0])
#<null>
$23 = void
(gdb) p jl_(*'jl_sym#curly5')
#<null>
$24 = void
(gdb) p jl_(args[1])
:parameters
$25 = void
(gdb) p jl_(*'jl_sym#Type6')
:parameters
$26 = void

Given this works with release 0.4, I think this is an issue with how we feed LLVM with symbol addresses/values.

[vtjnash]

If you run the debug binary, the SSP shows really quickly what is wrong: llvm is emitting a triple-dereference for all globals (GOT + Global + Garbage), while it should only be doing the first two.

exactly the mistake we are making in codegen.

llvm global variables are uniformly treated as pointers and it is not legal to try to take their address (nor any way to express that operations), whereas gdb / C automatically dereferences them to give their value when they are used (and & retrieves the address).

[yuyichao]

Relavant ASM. https://gist.github.com/yuyichao/83111d04eafadd03c03f return address is 0xb6ff60e8

[yuyichao]

Hmmm. Bisect points to 594c983 as the first bad commit so it seems that FastISel is causing the issue.

594c9836af3f5fe05ee4307ae28011c4151f2fe0 is the first bad commit                                                                                                       
commit 594c9836af3f5fe05ee4307ae28011c4151f2fe0                                                                                                                        
Author: Keno Fischer <kfischer+github@college.harvard.edu>                                                                                                             
Date:   Mon Sep 14 17:04:36 2015 -0400                                                                                                                                 

    Enable fast ISel for MCJIT                                                                                                                                         

    This starts the long march back to acceptable codegen performance. By enabling fast-isel, I see about a 10-20% improvement in overall execution time when running the test suite. Also from what I've been told, generated code performance isn't significantly worse than                                                                
    with the regular regalloc, though this is something we should benchmark at some point.                                                                             

:040000 040000 bc9cf41b234a7aa199fd8ceb26b8c78dbdae7932 b08e56a1c9665827b5088c264afaf8c597b695ff M      src                                                            

yuyichao$ git bisect log
git bisect start
# bad: [d233ef3724511a5f60923ffe7f70f150545ca8b1] Merge pull request #13278 from JuliaLang/ksh/doclapack
git bisect bad d233ef3724511a5f60923ffe7f70f150545ca8b1
# good: [c47407f630edba30998ac956797bbf29e5f29195] Merge branch 'jn/makefile_o'
git bisect good c47407f630edba30998ac956797bbf29e5f29195
# good: [ef833f04017eb56f1832d58286cbccf93ad4267d] Merge pull request #13163 from ginggs/patch-1
git bisect good ef833f04017eb56f1832d58286cbccf93ad4267d
# bad: [48d5c7cbc901bed3410447ce3ea6e11a239a2ddc] Merge pull request #13212 from ssfrr/abstractarray_similar_doc
git bisect bad 48d5c7cbc901bed3410447ce3ea6e11a239a2ddc
# good: [b1e447f1f75bd51e3091470b3e3582854e61da8d] Merge pull request #13062 from JuliaLang/ird/testnext
git bisect good b1e447f1f75bd51e3091470b3e3582854e61da8d
# bad: [db0f9432d10cb2af3ab920dd1a8342e282096a1a] Merge pull request #13169 from JuliaLang/teh/snoop
git bisect bad db0f9432d10cb2af3ab920dd1a8342e282096a1a
# bad: [a3252e0ae7819e20e7b4130b4b7be50ee4f9d03f] Merge pull request #13128 from JuliaLang/kf/fastisel
git bisect bad a3252e0ae7819e20e7b4130b4b7be50ee4f9d03f
# good: [2c59bc12348c054a806f015d3bdf391ec487f1ed] Merge pull request #13245 from JuliaLang/mn/12679
git bisect good 2c59bc12348c054a806f015d3bdf391ec487f1ed
# good: [53ec5499f05f5f2cd45881eafff1cf20537686f6] Merge pull request #13217 from catawbasam/basedoc4
git bisect good 53ec5499f05f5f2cd45881eafff1cf20537686f6
# bad: [594c9836af3f5fe05ee4307ae28011c4151f2fe0] Enable fast ISel for MCJIT
git bisect bad 594c9836af3f5fe05ee4307ae28011c4151f2fe0
# first bad commit: [594c9836af3f5fe05ee4307ae28011c4151f2fe0] Enable fast ISel for MCJIT

A quick Google search didn't find anything suspicious about FastISel on ARM but as @vtjnash pointed out, the global variables are correct but the instructions emitted are wrong.

[yuyichao]

@Keno

[vtjnash]

we tracked this down to FastISel picking the wrong instruction sequences. apparently this is a known, old bug in llvm (https://llvm.org/bugs/show_bug.cgi?id=16323). FastISel appears to be forcibly disabled on llvm-svn as of http://llvm.org/viewvc/llvm-project?view=revision&revision=248027, which explains why it worked there

[yuyichao]

They probably still have some other bug fixes because the commit only disable FastISel for pre-v6 and AFAIK I was testing is armv7 with JULIA_CPU_TARGET=cortex-a9 .

[vtjnash]

not sure then, there may have been some other nearby commits with similar effects. it does seem that processor should have v6Ops per https://github.com/llvm-mirror/llvm/blob/9cc82f0382e6fa5d35c4fbb0a18c1106dab1d7a8/lib/Target/ARM/ARM.td#L388-L391 (fwiw, I was not setting the JULIA_CPU_TARGET in my tests, so I did not have v6Ops available)

[yuyichao]

Assuming it's the same machine you took the last youtube video and the scaleway FAQ, the server should also be a quad-core cortex-a9 processor and LLVM should be able to recognize that with -C native.

[ViralBShah]

I was able to build with LLVM 3.6.1 and 3.7.0 on my chromebook, and everything seems to be the same as before.