Segfault on v10.7.0

[tlbdk]

E PID 8 received SIGSEGV for address: 0x0
E /app/node_modules/segfault-handler/build/Release/segfault-handler.node(+0x2c0d)[0x7f4e099bac0d]
E /lib/x86_64-linux-gnu/libpthread.so.0(+0x128e0)[0x7f4e0bd958e0]
E node[0xe837ac]
E node(_ZN2v88internal7Factory20NewStringFromTwoByteENS0_6VectorIKtEENS0_13PretenureFlagE+0x19)[0xe83c09]
E node(_ZN2v86String14NewFromTwoByteEPNS_7IsolateEPKtNS_13NewStringTypeEi+0x9a)[0xac57ba]
E node(_ZN4node11StringBytes6EncodeEPN2v87IsolateEPKtmPNS1_5LocalINS1_5ValueEEE+0x29d)[0x95aaad]
E node[0x95c9ed]
E node[0xb44409]
E node(_ZN2v88internal21Builtin_HandleApiCallEiPPNS0_6ObjectEPNS0_7IsolateE+0xb9)[0xb44f79]
E [0x1bb2750041bd]

• Version:
  node -v
  v10.7.0
• Platform:
  Linux d187343d97b6 4.9.93-linuxkit-aufs deps: update openssl to 1.0.1j #1 SMP Wed Jun 6 16:55:56 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[tlbdk]

Same issue on v10.9.0:

E PID 8 received SIGSEGV for address: 0x0
E /app/node_modules/segfault-handler/build/Release/segfault-handler.node(+0x2c0d)[0x7f4dfe980c0d]
E /lib/x86_64-linux-gnu/libpthread.so.0(+0x128e0)[0x7f4e00d5b8e0]
E node[0xe9eb00]
E node(_ZN2v88internal7Factory20NewStringFromTwoByteENS0_6VectorIKtEENS0_13PretenureFlagE+0x19)[0xe9ef49]
E node(_ZN2v86String14NewFromTwoByteEPNS_7IsolateEPKtNS_13NewStringTypeEi+0x9a)[0xadf7ca]
E node(_ZN4node11StringBytes6EncodeEPN2v87IsolateEPKtmPNS1_5LocalINS1_5ValueEEE+0x29d)[0x96cebd]
E node[0x96edfd]
E node[0xb5996f]
E node(_ZN2v88internal21Builtin_HandleApiCallEiPPNS0_6ObjectEPNS0_7IsolateE+0xb9)[0xb5a4d9]
E [0x30fa8a1dc01d]

[addaleax]

Can you share code that reproduces this? Do you use the official binaries downloaded from https://nodejs.org/, or from another source?

[tlbdk]

Sadly no, It's a rather large application that only crashes in the docker container running in staging and production environments, so the issues seem to be Linux specific or at least not happening on MacOSX where we are doing development.

We are using the official binaries you can see the docker images we use here: https://github.com/connectedcars/docker-node

Also, we are not seeing the issue on Node 8.x.

If you could give us some instructions on extracting more debug information, we can help with that?

[baslr]

@tlbdk oom? can you run node with gdb?

[addaleax]

The relevant symbol here is 0x96edfd which is inside node::(anonymous namespace)::DecodeData(v8::FunctionCallbackInfo<v8::Value> const&), which is (unfortunately) not provided by segfault-handler.

How consistently are you seeing these issues? Is there a chance you could run modified node binaries, or possibly even compile a debug build of Node?

[tlbdk]

@baslr Don't think it's an out of memory issue, we can run node with gdb, any instructions on how to do that?

[tlbdk]

@addaleax It seems pretty consistent, so it would be easy to run a modified node binary. If you have a prebuilt debug build of node that would be the easiest.

[addaleax]

@tlbdk I think the ideal situation would be causing a crash with a debug build to generate a core dump, then provide that debug binary + the core dump together (warning: these files are going to be huge).

If you want to run gdb yourself, you’d start gdb path/to/debug/binary and then enter run <node.js arguments> inside the shell. There, once the crash occurs, you could run bt to get a much better stack trace, use frame N to select individual frames from the stack as listed by bt, and p (expression) to print the values of variables etc.

[tlbdk]

@addaleax ok, what entails a debug build, do you have any instructions, should I just enable all the debug flags?

 ./configure | grep debug
                 'debug_nghttp2': 'false',
                 'node_debug_lib': 'false',
                 'v8_optimized_debug': 0,

[addaleax]

@tlbdk It’s ./configure --debug – what you’re looking at in that output seem to be feature flags, not compiler settings

[tlbdk]

@addaleax we finally managed to build a debug image to run the code and have captured a core dump, note that we sanitized it for passwords using:

perl -pi -e 's/password/xxxxxxxx/g' core.node.7.1535125297

Hope this is ok, else please provide another method for doing this.

Core dump: https://storage.googleapis.com/connectedcars-staging-public/node-debug/core.node.7.1535125297.gz
Debug node: https://storage.googleapis.com/connectedcars-staging-public/node-debug/node.debug.gz

[addaleax]

@tlbdk That doesn’t quite look like a debug build – those are typically in the range of ~600 MB binaries, whereas this one is the size of a regular Node binary…

./configure --debug generates two binaries, the regular one (node aka out/Release/node) and a debug build (node_g aka out/Debug/node).

[tlbdk]

@addaleax We just did ./configure --debug && make install

I now added:

cp -f out/Debug/node /opt/node-v10.9.0-linux-x64/bin/node
ls -lh /usr/local/bin/node
-rwxr-xr-x 1 root root 575M Aug 24 18:22 /usr/local/bin/node

Do I need to do anything else?

I would be nice with a short howto guide on how to do this, would be happy to write one if you had a good location to put it?

The build tasks 2-3 hours and hitting the issue another 10-20 minutes so we might not have a new core dump before Monday.

[addaleax]

@tlbdk That looks about right, yes. :)

Depending on how much you think a guide would be, BUILDING.md in this repo might be a good place?