segfault in TLS

[EricTheOne]

Reproduces 100% of the time within a minute of running the server. Server opens multiple TLS client connections (as a client).

I'll gladly help with testing a fix or sharing some of the backtrace data, but cannot share server code or specific scenario.

Please see nodejs/node-v0.x-archive#8780, I'm not 100% sure but I think it may be the same issue. I'm unable to reproduce it in node 0.12.2 (server is running fine, the segfault doesn't happen).

Backtrace (commit 7693705):

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000069eed0 in SSL_read ()
(gdb) backtrace
#0  0x000000000069eed0 in SSL_read ()
#1  0x0000000000d40a39 in node::TLSWrap::ClearOut() ()
#2  0x0000000000d4117e in node::TLSWrap::OnReadImpl(long, uv_buf_t const*, uv_handle_type, void*) ()
#3  0x0000000000ce109b in node::JSStream::ReadBuffer(v8::FunctionCallbackInfo<v8::Value> const&) ()
#4  0x0000000000850002 in v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) ()
#5  0x000000000087721b in v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) ()
#6  0x00002307b4e060bb in ?? ()
#7  0x00003c26fce690f9 in ?? ()
#8  0x00002307b4e06001 in ?? ()
#9  0x00007ffe6b7e47f0 in ?? ()
#10 0x00007ffe6b7e4850 in ?? ()
#11 0x00002307b4e99739 in ?? ()
#12 0x00002bb54efbee61 in ?? ()
#13 0x00003c26fce6b669 in ?? ()
...

dmesg:

segfault at 30 ip 000000000069eed0 sp 00007ffe6b7e41a8 error 4 in iojs[400000+d0e000]

Although nodejs/node-v0.x-archive#8780 contains a suggested fix, I don't see that it was applied in node.js 0.12.2, so either something else fixed it or the situation is different.

Naively applying the fix suggested in 8780 on io.js 2.0.1 (18d457b) doesn't seem to help. Instead I'm failing on an assertion.

"Fix" (tls_wrap.cc):

if (wrap->enc_out_) {
    NodeBIO::FromBIO(wrap->enc_out_)->Read(nullptr, wrap->write_size_);
    // Ensure that the progress will be made and `InvokeQueued` will be called.
    wrap->ClearIn();
}

Backtrace with fix applied:

Program terminated with signal SIGABRT, Aborted.
#0  0x00007fc17f768cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56  ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) backtrace
#0  0x00007fc17f768cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007fc17f76c0d8 in __GI_abort () at abort.c:89
#2  0x00007fc17f761b86 in __assert_fail_base (fmt=0x7fc17f8b2830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0xf7d1f9 "(bio->ptr) != (nullptr)", file=file@entry=0xf7d1e0 "../src/node_crypto_bio.h", line=line@entry=77, 
    function=function@entry=0xf7e080 <_ZZN4node7NodeBIO7FromBIOEP6bio_stE19__PRETTY_FUNCTION__> "static node::NodeBIO* node::NodeBIO::FromBIO(BIO*)") at assert.c:92
#3  0x00007fc17f761c32 in __GI___assert_fail (assertion=0xf7d1f9 "(bio->ptr) != (nullptr)", file=0xf7d1e0 "../src/node_crypto_bio.h", line=77, 
    function=0xf7e080 <_ZZN4node7NodeBIO7FromBIOEP6bio_stE19__PRETTY_FUNCTION__> "static node::NodeBIO* node::NodeBIO::FromBIO(BIO*)") at assert.c:101
#4  0x0000000000673989 in node::NodeBIO::FromBIO(bio_st*) [clone .isra.22] [clone .part.23] ()
#5  0x0000000000d409bf in node::TLSWrap::EncOutCb(node::WriteWrap*, int) ()
#6  0x0000000000850002 in v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) ()
#7  0x000000000087721b in v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) ()
#8  0x00003993517060bb in ?? ()
#9  0x00007fff1da78730 in ?? ()
#10 0x0000399351706001 in ?? ()
#11 0x00007fff1da78710 in ?? ()
#12 0x00007fff1da78778 in ?? ()
#13 0x0000399351a8292c in ?? ()
#14 0x00001507279beb79 in ?? ()
#15 0x0000000000000000 in ?? ()

[EricTheOne]

@indutny

[indutny]

May I ask you to give a try to the following patch #1702 ?

[EricTheOne]

@indutny gladly and thanks for the quick response. Applied the patch on io.js master (4e2f999), still getting segfault:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000069eed0 in SSL_read ()
(gdb) backtrace
#0  0x000000000069eed0 in SSL_read ()
#1  0x0000000000d40ae9 in node::TLSWrap::ClearOut() ()
#2  0x0000000000d4122e in node::TLSWrap::OnReadImpl(long, uv_buf_t const*, uv_handle_type, void*) ()
#3  0x0000000000ce109b in node::JSStream::ReadBuffer(v8::FunctionCallbackInfo<v8::Value> const&) ()
#4  0x0000000000850002 in v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) ()
#5  0x000000000087721b in v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) ()
#6  0x00000abcd52060bb in ?? ()
#7  0x00000abcd5206001 in ?? ()
#8  0x00007fffb979c580 in ?? ()
#9  0x00007fffb979c5d0 in ?? ()
#10 0x00000abcd55a6f1a in ?? ()
#11 0x000030f334efeea9 in ?? ()
#12 0x000015aab64b2489 in ?? ()
#13 0x00001bd0488c0841 in ?? ()
#14 0x000030f334efeea9 in ?? ()
#15 0x00000b493ed87e89 in ?? ()
#16 0x00001bd04889a081 in ?? ()
#17 0x00007fffb979c6a8 in ?? ()
#18 0x00000abcd5296468 in ?? ()
#19 0x000015aab64b2489 in ?? ()
#20 0x00001bd0488addf1 in ?? ()
#21 0x00000abcd54c8824 in ?? ()
#22 0x00000abcd54c8824 in ?? ()
#23 0x00003d23e40be3c1 in ?? ()
#24 0x00002f4a2ebf7a29 in ?? ()
#25 0x00000a9d72d66921 in ?? ()
#26 0x000015aab64b2489 in ?? ()
#27 0x000011d9d60bb679 in ?? ()
#28 0x00000a9d72dab0a9 in ?? ()
#29 0x00000a9d72d8dcf1 in ?? ()
#30 0x00007fff00000001 in ?? ()
#31 0x00000abc00000002 in ?? ()
#32 0x00007fffb979c700 in ?? ()
#33 0x000011d9d60bb631 in ?? ()
#34 0x00000a9d72d04141 in ?? ()
#35 0x00000a9d72d04161 in ?? ()
#36 0x00000b493ed87e89 in ?? ()
#37 0x00000a9d72d04101 in ?? ()
#38 0x00001bd0488c06f9 in ?? ()
#39 0x00000a9d72d04161 in ?? ()
#40 0x00000a9d72dc9181 in ?? ()
#41 0x000011d9d607ccf9 in ?? ()
#42 0x00000a9d72dc9181 in ?? ()
#43 0x000011d9d607ccf9 in ?? ()
#44 0x00007fffb979c6e0 in ?? ()
#45 0x00000abcd5214b35 in ?? ()
#46 0x000012917282dcf1 in ?? ()
#47 0x00001bd0488addf1 in ?? ()
#48 0x0000000200000000 in ?? ()
#49 0x00000a9d72dc9181 in ?? ()
#50 0x0000000a00000000 in ?? ()
#51 0x00007fffb979c778 in ?? ()
#52 0x00000abcd5621a9f in ?? ()
#53 0x000015aab64b2489 in ?? ()
#54 0x000012917282dcf1 in ?? ()
#55 0x00001bd0488addf1 in ?? ()
#56 0x00000a9d72d04101 in ?? ()
#57 0x00007fffb979c758 in ?? ()
#58 0x000030f334e1c389 in ?? ()
#59 0x00007fffb979c800 in ?? ()
#60 0x0000000000ac2c48 in v8::internal::Isolate::RunMicrotasks() ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

[EricTheOne]

By the way, as I was testing the patch npm seemed to fail installing four different modules.
I'm running configure && make && make install on the io.js repo, anything I'm missing to get npm to work?

> heapdump@0.3.5 install /.../node_modules/heapdump
> node-gyp rebuild

gyp WARN install got an error, rolling back install
gyp ERR! configure error 
gyp ERR! stack Error: 404 status code downloading tarball
gyp ERR! stack     at Request.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/install.js:246:14)
gyp ERR! stack     at emitOne (events.js:82:20)
gyp ERR! stack     at Request.emit (events.js:169:7)
gyp ERR! stack     at Request.onRequestResponse (/usr/local/lib/node_modules/npm/node_modules/request/request.js:1156:10)
gyp ERR! stack     at emitOne (events.js:77:13)
gyp ERR! stack     at ClientRequest.emit (events.js:169:7)
gyp ERR! stack     at HTTPParser.parserOnIncomingClient (_http_client.js:414:21)
gyp ERR! stack     at HTTPParser.parserOnHeadersComplete (_http_common.js:88:23)
gyp ERR! stack     at TLSSocket.socketOnData (_http_client.js:304:20)
gyp ERR! stack     at emitOne (events.js:77:13)
gyp ERR! System Linux 3.xx.x-xx-generic
gyp ERR! command "/usr/local/bin/iojs" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /.../node_modules/heapdump
gyp ERR! node -v v2.0.2
gyp ERR! node-gyp -v v1.0.3
gyp ERR! not ok 

[EricTheOne]

@indutny I've tested the changes on 2.1.0 as well, still segfaults:

#0  0x000000000069f060 in SSL_read ()
#1  0x0000000000d31499 in node::TLSWrap::ClearOut() ()
#2  0x0000000000d31bce in node::TLSWrap::OnReadImpl(long, uv_buf_t const*, uv_handle_type, void*) ()
#3  0x0000000000cd285b in node::JSStream::ReadBuffer(v8::FunctionCallbackInfo<v8::Value> const&) ()
#4  0x000000000084d342 in v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) ()
#5  0x000000000087457b in v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) ()

Any way I can help with more info?

[indutny]

@EricTheOne I commented about it on other issue. I really need a test case that reproduces the problem. It doesn't look like I fully understand the problem atm.