Segfault when aborting matrix multiplication

[andreasnoack]

I get this systematically

julia> mA=randn(5000,5000);mB=randn(5000,5000);

julia> mA*mB
^CSegmentation fault: 11

when I try to abort the computation. Julia runs in the terminal on a MacBook Pro with Mountain Lion. Shouldn't it be okay to abort a matrix multiplication like this or are crashes expected when aborting?

[staticfloat]

I get this:

julia> mA*mB
^C
 in gemm! at blas.jl:267
 in gemm_wrapper at matmul.jl:276
 in gemm_wrapper at matmul.jl:265
 in * at matmul.jl:84

julia> Segmentation fault: 11

And no, I do not believe this should be happening. I seem to remember Stefan saying "Any segfault is automatically a bug". :)

[ViralBShah]

During that call, the control is actually with openblas. Typically, you would intercept the signal, and do a cleanup in the handler. However, many of the native libraries are not designed with this kind of interactive usage in mind. It would still be nice to not have a segfault, if possible.

[JeffBezanson]

Short answer: just use linux :-P

We can look into it, but it's unlikely we'll be able to make ctl-C work perfectly in every case. Once an async signal happens the process is technically in an ill-defined state.

[staticfloat]

Does OpenBLAS reset signal handlers on function calls? If not, shouldn't Julia's signal handlers be getting called no matter where execution is at the time of the signal?

Also, the segfault doesn't always happen immediately for me, for instance in the example above, Julia was able to print out the stack trace, and ask for another prompt before segfaulting. My guess is this is a memory freeing bug triggered by the gc. Some further digging reveals:

julia> mA=randn(5000,5000);mB=randn(5000,5000);

julia> mA*mB
^C

julia> 

julia> 2+2
4

julia> gc()
Segmentation fault: 11

[staticfloat]

I managed to capture this in gdb:

julia> mA=randn(5000,5000);mB=randn(5000,5000);

julia> mA*mB
^C
Program received signal SIGINT, Interrupt.
0x0000000102e925f3 in .L12 ()
(gdb) signal SIGINT
Continuing with signal SIGINT.


julia> 
Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x000000012d43e5e0
[Switching to process 38681 thread 0x1203]
0x0000000102e92895 in .L18 ()
(gdb) bt
#0  0x0000000102e92895 in .L18 ()
#1  0x00000000000004e2 in ?? ()
(gdb) info threads
  3                         0x00007fff8e5ac0fa in __psynch_cvwait ()
* 2                         0x0000000102e92895 in .L18 ()
  1 "com.apple.main-thread" 0x00007fff8e5ac322 in select$DARWIN_EXTSN ()
(gdb) thread 3
[Switching to thread 3 (process 38681)]
0x00007fff8e5ac0fa in __psynch_cvwait ()
(gdb) bt
#0  0x00007fff8e5ac0fa in __psynch_cvwait ()
#1  0x00007fff92de7f89 in _pthread_cond_wait ()
#2  0x0000000100082d7b in run_io_thr ()
#3  0x00007fff92de3742 in _pthread_start ()
#4  0x00007fff92dd0181 in thread_start ()
(gdb) thread 1
[Switching to thread 1 (process 38681), "com.apple.main-thread"]
0x00007fff8e5ac322 in select$DARWIN_EXTSN ()
(gdb) bt
#0  0x00007fff8e5ac322 in select$DARWIN_EXTSN ()
#1  0x00007fff8e5ab20f in select ()
#2  0x0000000101d260eb in ?? ()
#3  0x0000000101d25987 in ?? ()
#4  0x0000000100049c75 in jl_apply_generic ()
#5  0x0000000101d25255 in ?? ()
#6  0x000000010008480e in start_task ()
#7  0x00000001000839ca in julia_trampoline ()
#8  0x0000000100001b7b in main ()
(gdb) 

It paused at the julia prompt for a couple seconds (I was going to type gc() but it crashed before I could), so hopefully these stack traces are more focused on the issue at hand.

[andreasnoack]

I have to push a little harder on my Ubuntu but it can do similar tricks

julia> mA=randn(5000,5000);mB=randn(5000,5000);

julia> mA*mB
^C

julia> mA*mB
^C

julia> mA*mB
BLAS : Program is Terminated. Because you tried to allocate too many memory regions.
Segmentation fault (core dumped)

I is not a big deal for me but I did not recall such behaviour when working with R and thought there might be a fix.

[ViralBShah]

In Matlab, you don't typically get segfaults when you do such stuff. Usually, matlab will not let you do a ctrl-c when it is in running native code. Not sure how R handles all this gracefully.

[StefanKarpinski]

We should definitely take a look at what R does. They may have some good tricks to copy.

[staticfloat]

I guess I don't understand why CTRL^C'ing in OpenBLAS code should give us problems vs. being in Julia code when we CTRL^C.

Usually, matlab will not let you do a ctrl-c when it is in running native code

MATLAB queues the CTRL^C until you're back in MATLAB code, and then breaks.

[JeffBezanson]

We have the ability to defer SIGINT around blocks of code, and it is used in some places in the run time system. We could expose this as a macro. It would be silly to use it around every ccall though.

[staticfloat]

I thought it was pretty standard procedure to have signal handlers setup at all times, so that no matter what happens signal-wise you aren't either interpreting signals as SIGTERM when you don't want to, or even worse, ignoring signals that should be having an effect. Having to wrap something around every ccall sounds like the wrong way to do things, but shouldn't Julia provide a signal handler that at the very least gracefully exits on signals? (Again, I don't think I'm understanding why you would segfault if the SIGINT happens during OpenBLAS execution vs. not segfaulting during Julia execution.)

[JeffBezanson]

We do have handlers set up, we just defer the handling in some cases so that certain actions are atomic. But in any case it is probably still possible for SIGINT to leave the process in an inconsistent state.

I just tried this example on linux, and using openblas threads I got a segfault. After setting OPENBLAS_NUM_THREADS=1 the segfault went away. This is 100% reproducible for me --- threads = segfault, 1 thread = works. So something in how the child threads are handled seems to be implicated. Do others see the same thing?

[andreasnoack]

I see the same thing on my Mac. Single threaded works.

[vtjnash]

From your comments, Jeff, it sounds like the openblas child threads are continuing happily along on whatever they were doing, unaware that the SIGINT has jumped execution out of openblas control (and that its temporary arrays will soon be GC'd). Unless there is a way to gracefully (or not) tell openblas to abort everything, perhaps calls to blas functions should be atomic to interruption by signals?

[ViralBShah]

Cc: @xianyi