Segmentation fault of node 10.8.0 x64 on Fedora 27 when running from `PATH`

[rocketraman]

• Version:

node --version
v10.8.0

• Platform:

$ cat /etc/os-release
NAME=Fedora
VERSION="27 (Twenty Seven)"
ID=fedora
VERSION_ID=27
PRETTY_NAME="Fedora 27 (Twenty Seven)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:fedoraproject:fedora:27"
HOME_URL="https://fedoraproject.org/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=27
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=27
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"

$ uname -a
Linux edison 4.17.12-100.fc27.x86_64 #1 SMP Fri Aug 3 15:00:33 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

• Subsystem: Node

I get a segfault just by running node -- no arguments, just trying to get an interactive shell:

$ type node
node is /opt/node-v10.8.0-linux-x64/bin/node

$ node
fish: “node” terminated by signal SIGSEGV (Address boundary error)

Here is the output from coredumpctl:

           PID: 91027 (node)
           UID: 1000 (raman)
           GID: 1000 (raman)
        Signal: 11 (SEGV)
     Timestamp: Tue 2018-08-14 12:06:28 EDT (5s ago)
  Command Line: node
    Executable: /opt/node-v10.8.0-linux-x64/bin/node
 Control Group: /user.slice/user-1000.slice/session-2.scope
          Unit: session-2.scope
         Slice: user-1000.slice
       Session: 2
     Owner UID: 1000 (raman)
       Boot ID: 6d5855baaf2c4f51ad2472a1a4631ae8
    Machine ID: ced7a5bf77e545c28c0ba7b313e98ca1
      Hostname: edison
       Storage: /var/lib/systemd/coredump/core.node.1000.6d5855baaf2c4f51ad2472a1a4631ae8.91027.1534262788000000.lz4
       Message: Process 91027 (node) of user 1000 dumped core.
                
                Stack trace of thread 91027:
                #0  0x00007ff3938b8c5a _ZN2v816FunctionTemplate3NewEPNS_7IsolateEPFvRKNS_20FunctionCallbackInfoINS_5ValueEEEENS_5LocalIS4_EENSA_INS_9SignatureEEEi (libnode.so)
                #1  0x00007ff393e52481 _ZN4node17ContextifyContext4InitEPNS_11EnvironmentEN2v85LocalINS3_6ObjectEEE (libnode.so)
                #2  0x00007ff393e52443 _ZN4node14InitContextifyEN2v85LocalINS0_6ObjectEEENS1_INS0_5ValueEEENS1_INS0_7ContextEEE (libnode.so)
                #3  0x00000000008baff3 _ZN4nodeL10GetBindingERKN2v820FunctionCallbackInfoINS0_5ValueEEE (node)
                #4  0x0000000000b4ee49 _ZN2v88internal12_GLOBAL__N_119HandleApiCallHelperILb0EEENS0_11MaybeHandleINS0_6ObjectEEEPNS0_7IsolateENS0_6HandleINS0_10HeapObjectEEESA_NS8_INS0_20FunctionTemplateInfoEEENS8_IS4_EENS0_16B
                #5  0x0000000000b4f9b9 _ZN2v88internal21Builtin_HandleApiCallEiPPNS0_6ObjectEPNS0_7IsolateE (node)
                #6  0x00001fb8db2841bd n/a (n/a)
                #7  0x00001fb8db293a09 n/a (n/a)
                #8  0x00001fb8db293a09 n/a (n/a)
                #9  0x00001fb8db290f95 n/a (n/a)
                #10 0x00001fb8db28a4a1 n/a (n/a)
                #11 0x0000000000e597f3 _ZN2v88internal9Execution4CallEPNS0_7IsolateENS0_6HandleINS0_6ObjectEEES6_iPS6_ (node)
                #12 0x0000000000ae84b3 _ZN2v88Function4CallENS_5LocalINS_7ContextEEENS1_INS_5ValueEEEiPS5_ (node)
                #13 0x00000000008be828 _ZN4node15LoadEnvironmentEPNS_11EnvironmentE (node)
                #14 0x00000000008c2c96 _ZN4node5StartEPN2v87IsolateEPNS_11IsolateDataEiPKPKciS8_ (node)
                #15 0x00000000008c191f _ZN4node5StartEiPPc (node)
                #16 0x00007ff396579f2a __libc_start_main (libc.so.6)
                #17 0x000000000087dbe5 _start (node)

This is running node directly from the tar.xz download on nodejs.org (sha256 sum verified).

Node 8 and 9 work without any issues.

[rocketraman]

Just tried node-v10.8.1-nightly2018081382830a809b-linux-x64, and it works fine. So it seems this issue was already fixed somewhere along the way.

[rocketraman]

Update, ok this is super-strange... never mind my last comment, the nightly has the same issue. It appears that the problem happens only when node is executed via a directory in the PATH:

Works fine when running node with the full path specified:

$ strace -e trace=file /opt/node-v10.8.0-linux-x64/bin/node
execve("/opt/node-v10.8.0-linux-x64/bin/node", ["/opt/node-v10.8.0-linux-x64/bin/"...], 0x7ffcae816b30 /* 125 vars */) = 0
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/opt/node-v10.8.0-linux-x64/bin/node", O_RDONLY|O_CLOEXEC) = 9
openat(AT_FDCWD, "/dev/urandom", O_RDONLY|O_NOCTTY|O_NONBLOCK) = 9
readlink("/proc/self/exe", "/opt/node-v10.8.0-linux-x64/bin/"..., 8191) = 36
readlink("/proc/self/fd/1", "/dev/pts/16", 255) = 11
stat("/dev/pts/16", {st_mode=S_IFCHR|0600, st_rdev=makedev(136, 16), ...}) = 0
openat(AT_FDCWD, "/dev/pts/16", O_RDWR|O_CLOEXEC) = 9
openat(AT_FDCWD, "/dev/null", O_RDONLY|O_CLOEXEC) = 10
readlink("/proc/self/fd/2", "/dev/pts/16", 255) = 11
stat("/dev/pts/16", {st_mode=S_IFCHR|0600, st_rdev=makedev(136, 16), ...}) = 0
openat(AT_FDCWD, "/dev/pts/16", O_RDWR|O_CLOEXEC) = 11
getcwd("/home/raman/tmp", 4096)         = 16
getcwd("/home/raman/tmp", 4096)         = 16
readlink("/proc/self/fd/0", "/dev/pts/16", 255) = 11
stat("/dev/pts/16", {st_mode=S_IFCHR|0600, st_rdev=makedev(136, 16), ...}) = 0
openat(AT_FDCWD, "/dev/pts/16", O_RDWR|O_CLOEXEC) = 12
> +++ exited with 0 +++

But borks when running the same binary from the PATH:

$ strace -e trace=file node
execve("/opt/node-v10.8.0-linux-x64/bin/node", ["node"], 0x7ffd6ef99d30 /* 125 vars */) = 0
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 9
openat(AT_FDCWD, "/lib64/libnode.so", O_RDONLY|O_CLOEXEC) = 9
openat(AT_FDCWD, "/dev/urandom", O_RDONLY|O_NOCTTY|O_NONBLOCK) = 9
readlink("/proc/self/exe", "/opt/node-v10.8.0-linux-x64/bin/"..., 8191) = 36
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x25108dda45} ---
+++ killed by SIGSEGV (core dumped) +++
fish: “strace -e trace=file node” terminated by signal SIGSEGV (Address boundary error)

Also, it works fine as root, but not as an unprivileged user. Selinux does not appear to be the culprit.

[richardlau]

               Stack trace of thread 91027:
               #0  0x00007ff3938b8c5a _ZN2v816FunctionTemplate3NewEPNS_7IsolateEPFvRKNS_20FunctionCallbackInfoINS_5ValueEEEENS_5LocalIS4_EENSA_INS_9SignatureEEEi (libnode.so)

This is running node directly from the tar.xz download on nodejs.org (sha256 sum verified).

The tar.xz download from nodejs.org doesn't contain nor link to libnode.so so this looks like interference with another Node.js installation.

[rocketraman]

The tar.xz download from nodejs.org doesn't contain nor link to libnode.so so this looks like interference with another Node.js installation.

Ah, interesting. Do you have any idea why this might be happening? As you can see from the strace the right node binary is being executed. I confirm it does not link to libnode.so:

# ldd /opt/node-v10.8.0-linux-x64/bin/node
        linux-vdso.so.1 (0x00007fff845e3000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f573a53d000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f573a335000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f5739fad000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f5739c62000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f5739a4b000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f573982d000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f5739477000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f573a741000)

And this issue does not happen with either node 8 or 9, which are installed in exactly the same way as 10.

[rocketraman]

Attached is the output of LD_DEBUG=all node. It contains the following:

    120607:     file=libnode.so [0];  dynamically loaded by node [0]
    120607:     find library=libnode.so [0]; searching
    120607:      search cache=/etc/ld.so.cache
    120607:       trying file=/lib64/libnode.so

ld-debug.txt.gz

[rocketraman]

I found out that libnode.so was owned by atom-libs, which appears to have been a leftover package since I uninstalled atom a while ago. Removing this package has fixed the problem, but I still find it odd that node 10 attempts to dynamically load this shared lib, which node 8 and 9 did not, and only in some cases such as not running it with an absolute path. While this is solved from my perspective, it certainly seems like this could be some bad logic in node 10.

[richardlau]

At this point I'd cc @nodejs/platform-linux but we don't appear to have such a group 😞 .

[hmt]

On Fedora 28 I had the same issue. I have no clue where my libnode.so came from. But thanks for giving the full debug logs. Learnt something new about strace and coredumps. 👍

[rocketraman]

@hmt You can try

rpm -q --whatprovides /path/to/your/libnode.so

The path you can get from the output of LD_DEBUG=all node.

That might tell you something useful about which rpm it came from, and if it isn't a package on your system then it was probably installed manually at some point.

I still think that something is wrong with nodes internals though.

[Trott]

@nodejs/build Any thoughts/insights on this one?

[rvagg]

nope, and I agree, Node shouldn't be trying to load libnode if it's a stand-alone build. I'm trying to find something in the commit list for 10 that are not in 8 that might point to a change that might be implicated but nothing is standing out. There has been a little bit of work to make compiling Node as a shared library so perhaps it's related to that. 🤷‍♂️ @bnoordhuis any thoughts on where to look for this one?

[Trott]

If It's specific to Fedora and/or other Red Hat OSes, maybe @danbev or @lance might have an idea?

[danbev]

If It's specific to Fedora and/or other Red Hat OSes, maybe @danbev or @lance might have an idea?

Nothing I can think of without having a closer look I'm afraid. If I have time I'll try to take a look later this week but it might not be until next week as this is a short week for me (not working Friday).