Segfault on Node 10.x doesn't occur in 6.x

[jvictor85]

• Version: v10.15.2 (also happens with other v10.x versions)
• Platform: Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0 GCP x86_64)

Since last year I've been having issues switching to 10.x from 6.x because of rare crashes that give no error. I have several production servers with hundreds of users and it's been too difficult to reproduce with a local server so far because it seems to only happens under load and it's fairly rare.

There's a large amount of code that I can't really reduce for testing without affecting my users. Since I haven't had the time or ability to debug this properly I was hoping someone could point me in the right direction, or that this would be helpful to the developers if this is a bug in 10.x or v8.

There are no errors to report, but here are some of the recent segfaults in various syslogs that only happen when a server is on node 10:

Feb 12 21:56:29 EU1 kernel: [22262355.042532] node[20512]: segfault at 3d77ac406468 ip 0000000000efade2 sp 00007f263e259c10 error 4 in node[400000+1e90000]

Mar 13 19:39:09 SEA1 kernel: [2908566.872204] node[4145]: segfault at 23e4891a2fb0 ip 0000000000efa272 sp 00007fbbf4dd0c10 error 4 in node[400000+1e8f000]

Mar 16 08:59:55 SEA1 kernel: [3129393.630360] node[14805]: segfault at 1ac3c9db79c8 ip 0000000000efa272 sp 00007f10629f0c10 error 4 in node[400000+1e8f000]

Mar 16 20:25:29 USW1 kernel: [3173535.851715] node[31823]: segfault at 13aa4aac9a78 ip 0000000000efa272 sp 00007fdb85380c10 error 4 in node[400000+1e8f000]

Mar 17 00:26:56 USE2 kernel: [25489067.874929] node[17011]: segfault at 93788ef0108 ip 0000000000efade2 sp 00007fc14bffec10 error 4 in node[400000+1e90000]

Mar 19 22:10:11 USW1 kernel: [3438995.257871] node[11791]: segfault at 7b0370d05c8 ip 0000000000efa272 sp 00007f88d3403c10 error 4 in node[400000+1e8f000]

Mar 21 11:46:28 USW1 kernel: [3574361.032453] node[18756]: segfault at 10f0170f9b8 ip 0000000000efa272 sp 00007fdb9e281c10 error 4 in node (deleted)[400000+1e8f000]

Mar 27 18:55:30 USE2 kernel: [26419545.476970] node[21011]: segfault at 706d2e8f0b8 ip 0000000000efade2 sp 00007f2b59fadc10 error 4 in node[400000+1e90000]

Apr 1 20:39:24 SEA1 kernel: [319450.383166] node[8710]: segfault at 16f9cfd832b0 ip 0000000000efa272 sp 00007f7850aa2c10 error 4 in node[400000+1e8f000]

Apr 1 20:23:53 USE1 kernel: [26742046.491931] node[4466]: segfault at 3253e97d4310 ip 0000000000efa272 sp 00007f1b2fffec10 error 4 in node[400000+1e8f000]

Apr 2 14:58:52 EU1 kernel: [26470558.192840] node[27273]: segfault at 1bbcf96ba358 ip 0000000000efade2 sp 00007f1e95a03c10 error 4 in node[400000+1e90000]

The module versions have been different throughout the last year and on different servers but here is what is currently being used on USW1 as an example:

• mongodb@3.1.0-beta4
• ws@5.2.0

I haven't done much testing in node 8.x due to some major garbage collection delay which is another issue altogether. Similarly I haven't had success on version 11.x. I was able to reproduce the crash on my local windows machine once upon a time after days of experimentation on node 10, but not reliably, and I can't seem to get it to occur anymore. It's possible some newer changes have reduced the frequency of this crash because I remember it happening more often in 2018.

I've been mostly using the N version switcher to test different versions. Let me know if there is any other info I can provide or some way to look up these segfault addresses to narrow things down. Thanks!

[sam-github]

Have you collected core files? They can be used to get more information about the crash.

Can you add https://github.com/nodejs/node-report to your app? It will give you loggable text output on a crash, that will help a lot. Its builtin to node now, but for 10.x you'll have to add it to your env, its not hard: https://github.com/nodejs/node-report#usage

[addaleax]

Also, do you use native addons, and if so, which? (find node_modules -name '*.node' might help). As far as I can tell, the instruction pointers at the time of the crash refer to two places in the garbage collection code.

To me, that makes it seem like an issue where a core dump would be more important than node-report output, although even the former might be hard to work with because it’s likely to be some sort of memory corruption issue where it’s hard to figure out the cause from the crash site.

[jvictor85]

Thanks for the help! For now I tried enabling core dumps on one of the 10.x servers with ulimit.. assuming I did it right it could take days or weeks to see the crash though.

I forgot to mention that I use PM2 to restart the server process whenever it goes down, hopefully that doesn't matter.

As for native addons I'm not sure.. I ran that find command and it just shows:
node_modules/v8-profiler/build/profiler/v5.7.0/node-v48-linux-x64/profiler.node

That module isn't loaded in by default though so I'm assuming it isn't related to the crash. I have to manually load it during run-time, and have only used it in some rare situations where I needed to profile for CPU bottlenecks. The basic modules I utilize other than ws and mongodb are http, https, os, fs.

I'm not familiar with memory corruption issues, but let me know if there's anything else I can do that might help. I wondered if it might be something in the garbage collector.. I noticed that the GC in 10.x will have a rare delay spike of 2000-3000ms out of nowhere. On 6.x the sweep usually only takes 300ms or so, but they are more often.

The current flags used (on all node versions):
--max-old-space-size=2400 --nouse-idle-notification --expose-gc

[jvictor85]

I got one pretty quick! A core file from this crash:

Apr 6 16:35:05 USE1 kernel: [27160290.897495] node[5357]: segfault at 1f6b91e12128 ip 0000000000efa272 sp 00007fea19e45c10 error 4 in node[400000+1e8f000]

It's about 1gb in size (90mb compressed). I'd love for someone to take a look at it though I'm not sure the best way to send it, and I'd like to avoid posting it publicly if possible. I added an email to my profile if someone can get in touch :)

[bnoordhuis]

@jvictor85 Can you get the stack trace? gdb /path/to/node /path/to/core, then bt.

[jvictor85]

Thanks! I tried that and it said it loaded symbols from node though it just shows these addresses:

Program terminated with signal SIGTRAP, Trace/breakpoint trap.
#0  0x00001b50e646b619 in ?? ()
[Current thread is 1 (LWP 32374)]
(gdb) bt
#0  0x00001b50e646b619 in ?? ()
#1  0x00000005000001ba in ?? ()
#2  0x0000000000000000 in ?? ()

[jvictor85]

Apparently I had some core files from before and didn't even realize it. Here is the stack trace from the Feb 12th segfault above, it's one of 2 different types of crashes I get it seems - this one at addr 0000000000efade2 and it has some better symbols in it:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000efade2 in v8::internal::RememberedSetUpdatingItem<v8::internal::MajorNonAtomicMarkingState>::Process() ()
[Current thread is 1 (Thread 0x7f263e25a700 (LWP 20512))]
(gdb) bt
#0  0x0000000000efade2 in v8::internal::RememberedSetUpdatingItem<v8::internal::MajorNonAtomicMarkingState>::Process() ()
#1  0x0000000000ee266b in v8::internal::PointersUpdatingTask::RunInParallel() ()
#2  0x0000000000edfd0f in v8::internal::ItemParallelJob::Task::RunInternal() ()
#3  0x0000000000bc7166 in v8::internal::CancelableTask::Run() ()
#4  0x00000000009625b9 in node::BackgroundRunner(void*) ()
#5  0x00007f263ee546db in start_thread (arg=0x7f263e25a700) at pthread_create.c:463
#6  0x00007f263eb7d88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

This EU server is running v10.15.0.

[bnoordhuis]

Okay, so it's a crash during GC. That probably means memory corruption but unfortunately the real bug is usually not anywhere near the site of the crash, neither in time or space.

What does info maps print?

[jvictor85]

Sorry not familiar with gdb but when I run that it says Undefined info command: "maps".

If it's memory corruption during GC I'm guessing this isn't anything I can fix in my own code then.. If it helps I remember trying to isolate this last year and that removing the check if(typeof x === "object") in a distance calculation function reduced the frequency of crashes by a lot. It made no sense to me but it went from crashing every 10-60 minutes to how it is now, about once a week and only under load. I'm not sure I can even reproduce that frequent crash anymore though, it was so unpredictable that simply adding a line to output something to the console would prevent the crash from happening in some cases.

[bnoordhuis]

Sorry, I was thinking of info proc mappings but that doesn't work with core files, only with live images. It'd be useful to check if the backtrace is always the same, i.e., if the crash site is consistent.

Aside: --nouse-idle-notification is a bit of lore that survived from Ye Olden Days but it's a no-op nowadays, you don't need it.

[jvictor85]

Thanks I'll phase out that flag going forward. I tracked down some more core files and collected the backtraces below, along with the node version and date. The one for the SEA server I wasn't sure on the exact version used, so I included backtraces for both possibilities.

SA1 (Node v10.2.1) May 27 2018

Program terminated with signal SIGTRAP, Trace/breakpoint trap.
#0  0x0000275f5cc74c60 in ?? ()
[Current thread is 1 (Thread 0x7f1966f83740 (LWP 2979))]
(gdb) bt
#0  0x0000275f5cc74c60 in ?? ()
#1  0x00007fff661431a0 in ?? ()
#2  0x0000275f5c9b0d79 in ?? ()
etc...

USW1 (Node v10.4.0) June 9 2018

Program terminated with signal SIGTRAP, Trace/breakpoint trap.
#0  0x000017b9ddac0db9 in ?? ()
[Current thread is 1 (Thread 0x7ff9daa14740 (LWP 21738))]
(gdb) bt
#0  0x000017b9ddac0db9 in ?? ()
#1  0x00001bbbc8e822e1 in ?? ()
#2  0x00001bbbc8e822e1 in ?? ()
...

USE2 (Node v10.15.0) March 16 2019

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000efade2 in v8::internal::RememberedSetUpdatingItem<v8::internal::MajorNonAtomicMarkingState>::Process() ()
[Current thread is 1 (Thread 0x7fc14bfff700 (LWP 17011))]
(gdb) bt
#0  0x0000000000efade2 in v8::internal::RememberedSetUpdatingItem<v8::internal::MajorNonAtomicMarkingState>::Process() ()
#1  0x0000000000ee266b in v8::internal::PointersUpdatingTask::RunInParallel() ()
#2  0x0000000000edfd0f in v8::internal::ItemParallelJob::Task::RunInternal() ()
#3  0x0000000000bc7166 in v8::internal::CancelableTask::Run() ()
#4  0x00000000009625b9 in node::BackgroundRunner(void*) ()
#5  0x00007fc1519e86db in start_thread (arg=0x7fc14bfff700) at pthread_create.c:463
#6  0x00007fc15171188f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

USW2 (Node v10.13.0) November 18 2018

Program terminated with signal SIGTRAP, Trace/breakpoint trap.
#0  0x000021a9ad996aa4 in ?? ()
[Current thread is 1 (Thread 0x7f83a4de3740 (LWP 2306))]
(gdb) bt
#0  0x000021a9ad996aa4 in ?? ()
#1  0x00000000032b6ce0 in ?? ()
#2  0x0000000000000001 in ?? ()
#3  0x0000000000000000 in ?? ()

SEA1 (Node v10.15.?) February 17 2019

Node v10.15.0:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000efade2 in v8::internal::RememberedSetUpdatingItem<v8::internal::MajorNonAtomicMarkingState>::Process() ()
[Current thread is 1 (Thread 0x7f13fa7fc700 (LWP 10353))]
(gdb) bt
#0  0x0000000000efade2 in v8::internal::RememberedSetUpdatingItem<v8::internal::MajorNonAtomicMarkingState>::Process() ()
#1  0x0000000000ee266b in v8::internal::PointersUpdatingTask::RunInParallel() ()
#2  0x0000000000edfd0f in v8::internal::ItemParallelJob::Task::RunInternal() ()
#3  0x0000000000bc7166 in v8::internal::CancelableTask::Run() ()
#4  0x00000000009625b9 in node::BackgroundRunner(void*) ()
#5  0x00007f14010796db in start_thread (arg=0x7f13fa7fc700) at pthread_create.c:463
#6  0x00007f1400da288f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Node v10.15.1:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000efade2 in void v8::internal::BodyDescriptorApply<v8::internal::CallIterateBody, void, v8::internal::Map*, v8::internal::HeapObject*, int, v8::internal::PointersUpdatingVisitor*>(v8::internal::InstanceType, v8::internal::Map*, v8::internal::HeapObject*, int, v8::internal::PointersUpdatingVisitor*) [clone .constprop.755] ()
[Current thread is 1 (LWP 10353)]
(gdb) bt
#0  0x0000000000efade2 in void v8::internal::BodyDescriptorApply<v8::internal::CallIterateBody, void, v8::internal::Map*, v8::internal::HeapObject*, int, v8::internal::PointersUpdatingVisitor*>(v8::internal::InstanceType, v8::internal::Map*, v8::internal::HeapObject*, int, v8::internal::PointersUpdatingVisitor*) [clone .constprop.755] ()
#1  0x0000000000ee266b in v8::internal::MarkCompactCollector::~MarkCompactCollector() ()
#2  0x0000000000edfd0f in v8::internal::ItemParallelJob::Run(std::shared_ptr<v8::internal::Counters>) ()
#3  0x0000000000bc7166 in v8::internal::Cancelable::~Cancelable() ()
#4  0x00000000009625b9 in node::os::Initialize(v8::Local<v8::Object>, v8::Local<v8::Value>, v8::Local<v8::Context>) ()
#5  0x00007f14010796db in ?? ()
#6  0x0000000000000000 in ?? ()

UST1 (Node v10.15.0) February 28 2019

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000efade2 in v8::internal::RememberedSetUpdatingItem<v8::internal::MajorNonAtomicMarkingState>::Process() ()
[Current thread is 1 (Thread 0x7ff5a77fe700 (LWP 19333))]
(gdb) bt
#0  0x0000000000efade2 in v8::internal::RememberedSetUpdatingItem<v8::internal::MajorNonAtomicMarkingState>::Process() ()
#1  0x0000000000ee266b in v8::internal::PointersUpdatingTask::RunInParallel() ()
#2  0x0000000000edfd0f in v8::internal::ItemParallelJob::Task::RunInternal() ()
#3  0x0000000000bc7166 in v8::internal::CancelableTask::Run() ()
#4  0x00000000009625b9 in node::BackgroundRunner(void*) ()
#5  0x00007ff5ad9de6db in start_thread (arg=0x7ff5a77fe700) at pthread_create.c:463
#6  0x00007ff5ad70788f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Hopefully someone can make sense of these or find a pattern that leads to something. We would love to finally be able to switch to node 10 for good, performance is critical for us. Let me know if there's anything else I can do. Thanks!

[jvictor85]

Also just got a new one today:

USE1 (Node v10.15.2) April 12 2019

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000efa272 in v8::internal::RememberedSetUpdatingItem<v8::internal::MajorNonAtomicMarkingState>::Process() ()
[Current thread is 1 (Thread 0x7f4bd59e4700 (LWP 2001))]
(gdb) bt
#0  0x0000000000efa272 in v8::internal::RememberedSetUpdatingItem<v8::internal::MajorNonAtomicMarkingState>::Process() ()
#1  0x0000000000ee1afb in v8::internal::PointersUpdatingTask::RunInParallel() ()
#2  0x0000000000edf19f in v8::internal::ItemParallelJob::Task::RunInternal() ()
#3  0x0000000000bc65f6 in v8::internal::CancelableTask::Run() ()
#4  0x0000000000962bb9 in node::BackgroundRunner(void*) ()
#5  0x00007f4bd6ddf6db in start_thread (arg=0x7f4bd59e4700) at pthread_create.c:463
#6  0x00007f4bd6b0888f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Many of them seem to point to this same function, though on this server and a couple others it's at the efa272 address rather than efade2.

[gireeshpunathil]

the one thing what is worth mentioning is the presence of TRACE_EVENT0 in this routine that internally includes accessing the trace agent; and in the past we have seen exit race between the main thread and helpers wherein the trace agent was cleaned up while other threads has still pending work to do with.

To confirm, could you please dump the stack of every threads?

(gdb) info thr

then switch to each thread by thr 1 , thr 2 etc. and then issuing bt

[gireeshpunathil]

if the main thread is seen in the exit route with some advancement into cleanup actions, then we are looking at what I suspect.

[jvictor85]

Sure thing, here are all 11 threads for the latest core dump from above:

(gdb) info thr
  Id   Target Id         Frame
* 1    Thread 0x7f4bd59e4700 (LWP 2001) 0x0000000000efa272 in v8::internal::RememberedSetUpdatingItem<v8::internal::MajorNonAtomicMarkingState>::Process() ()
  2    Thread 0x7f4bd69e6700 (LWP 1999) 0x00007f4bd6b089ee in __GI_epoll_pwait (epfd=14, events=events@entry=0x7f4bd69e2e00,
    maxevents=maxevents@entry=1024, timeout=timeout@entry=-1, set=set@entry=0x0) at ../sysdeps/unix/sysv/linux/epoll_pwait.c:42
  3    Thread 0x7f4bd7f68700 (LWP 2004) 0x00007f4bd6de86d6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0,
    futex_word=0x24a71c0 <node::inspector::(anonymous namespace)::start_io_thread_semaphore>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  4    Thread 0x7f4bbffff700 (LWP 2005) 0x00007f4bd6de59f3 in futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x24a7fc8 <cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  5    Thread 0x7f4bbf7fe700 (LWP 2006) 0x00007f4bd6de59f3 in futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x24a7fcc <cond+44>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  6    Thread 0x7f4bbeffd700 (LWP 2007) 0x00007f4bd6de59f3 in futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x24a7fc8 <cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  7    Thread 0x7f4bbe7fc700 (LWP 2008) 0x00007f4bd6de59f3 in futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x24a7fc8 <cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  8    Thread 0x7f4bd61e5700 (LWP 2000) __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  9    Thread 0x7f4bd7f5b740 (LWP 1998) 0x0000000000efa07a in v8::internal::RememberedSetUpdatingItem<v8::internal::MajorNonAtomicMarkingState>::Process() ()
  10   Thread 0x7f4bd51e3700 (LWP 2002) 0x00007f4bd6de5b0f in futex_wake (private=<optimized out>, processes_to_wake=2147483647,
    futex_word=0x3b13d94) at ../sysdeps/unix/sysv/linux/futex-internal.h:231
  11   Thread 0x7f4bd49e2700 (LWP 2003) __GI___libc_free (mem=0x7f4bc81275c0) at malloc.c:3101

[Switching to thread 1 (Thread 0x7f4bd59e4700 (LWP 2001))]
#0  0x0000000000efa272 in v8::internal::RememberedSetUpdatingItem<v8::internal::MajorNonAtomicMarkingState>::Process() ()
(gdb) bt
#0  0x0000000000efa272 in v8::internal::RememberedSetUpdatingItem<v8::internal::MajorNonAtomicMarkingState>::Process() ()
#1  0x0000000000ee1afb in v8::internal::PointersUpdatingTask::RunInParallel() ()
#2  0x0000000000edf19f in v8::internal::ItemParallelJob::Task::RunInternal() ()
#3  0x0000000000bc65f6 in v8::internal::CancelableTask::Run() ()
#4  0x0000000000962bb9 in node::BackgroundRunner(void*) ()
#5  0x00007f4bd6ddf6db in start_thread (arg=0x7f4bd59e4700) at pthread_create.c:463
#6  0x00007f4bd6b0888f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[Switching to thread 2 (Thread 0x7f4bd69e6700 (LWP 1999))]
#0  0x00007f4bd6b089ee in __GI_epoll_pwait (epfd=14, events=events@entry=0x7f4bd69e2e00, maxevents=maxevents@entry=1024,
    timeout=timeout@entry=-1, set=set@entry=0x0) at ../sysdeps/unix/sysv/linux/epoll_pwait.c:42
42      ../sysdeps/unix/sysv/linux/epoll_pwait.c: No such file or directory.
(gdb) bt
#0  0x00007f4bd6b089ee in __GI_epoll_pwait (epfd=14, events=events@entry=0x7f4bd69e2e00, maxevents=maxevents@entry=1024,
    timeout=timeout@entry=-1, set=set@entry=0x0) at ../sysdeps/unix/sysv/linux/epoll_pwait.c:42
#1  0x0000000000a56b6f in uv__io_poll (loop=loop@entry=0x3b143e8, timeout=-1) at ../deps/uv/src/unix/linux-core.c:275
#2  0x0000000000a4691b in uv_run (loop=0x3b143e8, mode=UV_RUN_DEFAULT) at ../deps/uv/src/unix/core.c:370
#3  0x0000000000966508 in node::BackgroundTaskRunner::DelayedTaskScheduler::Start()::{lambda(void*)#1}::_FUN(void*) ()
#4  0x00007f4bd6ddf6db in start_thread (arg=0x7f4bd69e6700) at pthread_create.c:463
#5  0x00007f4bd6b0888f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[Switching to thread 3 (Thread 0x7f4bd7f68700 (LWP 2004))]
#0  0x00007f4bd6de86d6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0,
    futex_word=0x24a71c0 <node::inspector::(anonymous namespace)::start_io_thread_semaphore>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
205     ../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
(gdb) bt
#0  0x00007f4bd6de86d6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0,
    futex_word=0x24a71c0 <node::inspector::(anonymous namespace)::start_io_thread_semaphore>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x24a71c0 <node::inspector::(anonymous namespace)::start_io_thread_semaphore>, abstime=0x0)
    at sem_waitcommon.c:111
#2  0x00007f4bd6de87c8 in __new_sem_wait_slow (
    sem=sem@entry=0x24a71c0 <node::inspector::(anonymous namespace)::start_io_thread_semaphore>, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007f4bd6de8839 in __new_sem_wait (sem=sem@entry=0x24a71c0 <node::inspector::(anonymous namespace)::start_io_thread_semaphore>)
    at sem_wait.c:42
#4  0x0000000000a53dc2 in uv__sem_wait (sem=<optimized out>) at ../deps/uv/src/unix/thread.c:579
#5  uv_sem_wait (sem=0x24a71c0 <node::inspector::(anonymous namespace)::start_io_thread_semaphore>) at ../deps/uv/src/unix/thread.c:635
#6  0x00000000009c420a in node::inspector::(anonymous namespace)::StartIoThreadMain(void*) ()
#7  0x00007f4bd6ddf6db in start_thread (arg=0x7f4bd7f68700) at pthread_create.c:463
#8  0x00007f4bd6b0888f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[Switching to thread 4 (Thread 0x7f4bbffff700 (LWP 2005))]
#0  0x00007f4bd6de59f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x24a7fc8 <cond+40>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88      ../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
(gdb) bt
#0  0x00007f4bd6de59f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x24a7fc8 <cond+40>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x24a7f60 <mutex>, cond=0x24a7fa0 <cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=cond@entry=0x24a7fa0 <cond>, mutex=mutex@entry=0x24a7f60 <mutex>) at pthread_cond_wait.c:655
#3  0x0000000000a53fe9 in uv_cond_wait (cond=cond@entry=0x24a7fa0 <cond>, mutex=mutex@entry=0x24a7f60 <mutex>)
    at ../deps/uv/src/unix/thread.c:735
#4  0x0000000000a4171e in worker (arg=0x0) at ../deps/uv/src/threadpool.c:76
#5  0x00007f4bd6ddf6db in start_thread (arg=0x7f4bbffff700) at pthread_create.c:463
#6  0x00007f4bd6b0888f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[Switching to thread 5 (Thread 0x7f4bbf7fe700 (LWP 2006))]
#0  0x00007f4bd6de59f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x24a7fcc <cond+44>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88      in ../sysdeps/unix/sysv/linux/futex-internal.h
(gdb) bt
#0  0x00007f4bd6de59f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x24a7fcc <cond+44>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x24a7f60 <mutex>, cond=0x24a7fa0 <cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=cond@entry=0x24a7fa0 <cond>, mutex=mutex@entry=0x24a7f60 <mutex>) at pthread_cond_wait.c:655
#3  0x0000000000a53fe9 in uv_cond_wait (cond=cond@entry=0x24a7fa0 <cond>, mutex=mutex@entry=0x24a7f60 <mutex>)
    at ../deps/uv/src/unix/thread.c:735
#4  0x0000000000a4171e in worker (arg=0x0) at ../deps/uv/src/threadpool.c:76
#5  0x00007f4bd6ddf6db in start_thread (arg=0x7f4bbf7fe700) at pthread_create.c:463
#6  0x00007f4bd6b0888f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[Switching to thread 6 (Thread 0x7f4bbeffd700 (LWP 2007))]
#0  0x00007f4bd6de59f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x24a7fc8 <cond+40>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88      in ../sysdeps/unix/sysv/linux/futex-internal.h
(gdb) bt
#0  0x00007f4bd6de59f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x24a7fc8 <cond+40>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x24a7f60 <mutex>, cond=0x24a7fa0 <cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=cond@entry=0x24a7fa0 <cond>, mutex=mutex@entry=0x24a7f60 <mutex>) at pthread_cond_wait.c:655
#3  0x0000000000a53fe9 in uv_cond_wait (cond=cond@entry=0x24a7fa0 <cond>, mutex=mutex@entry=0x24a7f60 <mutex>)
    at ../deps/uv/src/unix/thread.c:735
#4  0x0000000000a4171e in worker (arg=0x0) at ../deps/uv/src/threadpool.c:76
#5  0x00007f4bd6ddf6db in start_thread (arg=0x7f4bbeffd700) at pthread_create.c:463
#6  0x00007f4bd6b0888f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[Switching to thread 7 (Thread 0x7f4bbe7fc700 (LWP 2008))]
#0  0x00007f4bd6de59f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x24a7fc8 <cond+40>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88      in ../sysdeps/unix/sysv/linux/futex-internal.h
(gdb) bt
#0  0x00007f4bd6de59f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x24a7fc8 <cond+40>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x24a7f60 <mutex>, cond=0x24a7fa0 <cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=cond@entry=0x24a7fa0 <cond>, mutex=mutex@entry=0x24a7f60 <mutex>) at pthread_cond_wait.c:655
#3  0x0000000000a53fe9 in uv_cond_wait (cond=cond@entry=0x24a7fa0 <cond>, mutex=mutex@entry=0x24a7f60 <mutex>)
    at ../deps/uv/src/unix/thread.c:735
#4  0x0000000000a4171e in worker (arg=0x0) at ../deps/uv/src/threadpool.c:76
#5  0x00007f4bd6ddf6db in start_thread (arg=0x7f4bbe7fc700) at pthread_create.c:463
#6  0x00007f4bd6b0888f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[Switching to thread 8 (Thread 0x7f4bd61e5700 (LWP 2000))]
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
135     ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.
(gdb) bt
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f4bd6de2023 in __GI___pthread_mutex_lock (mutex=0x3b13d58) at ../nptl/pthread_mutex_lock.c:78
#2  0x0000000000a539c9 in uv_mutex_lock (mutex=<optimized out>) at ../deps/uv/src/unix/thread.c:285
#3  0x0000000000962bc1 in node::BackgroundRunner(void*) ()
#4  0x00007f4bd6ddf6db in start_thread (arg=0x7f4bd61e5700) at pthread_create.c:463
#5  0x00007f4bd6b0888f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[Switching to thread 9 (Thread 0x7f4bd7f5b740 (LWP 1998))]
#0  0x0000000000efa07a in v8::internal::RememberedSetUpdatingItem<v8::internal::MajorNonAtomicMarkingState>::Process() ()
(gdb) bt
#0  0x0000000000efa07a in v8::internal::RememberedSetUpdatingItem<v8::internal::MajorNonAtomicMarkingState>::Process() ()
#1  0x0000000000ee1afb in v8::internal::PointersUpdatingTask::RunInParallel() ()
#2  0x0000000000edf19f in v8::internal::ItemParallelJob::Task::RunInternal() ()
#3  0x0000000000edfcd7 in v8::internal::ItemParallelJob::Run(std::shared_ptr<v8::internal::Counters>) ()
#4  0x0000000000eebe68 in v8::internal::MarkCompactCollector::UpdatePointersAfterEvacuation() ()
#5  0x0000000000ef68d4 in v8::internal::MarkCompactCollector::Evacuate() ()
#6  0x0000000000ef7342 in v8::internal::MarkCompactCollector::CollectGarbage() ()
#7  0x0000000000ecf191 in v8::internal::Heap::MarkCompact() ()
#8  0x0000000000ecf881 in v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) ()
#9  0x0000000000ed0484 in v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) ()
#10 0x0000000000ed18bf in v8::internal::Heap::HandleGCRequest() ()
#11 0x0000000000e70c3a in v8::internal::StackGuard::HandleInterrupts() ()
#12 0x000000000113bde5 in v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*) ()
#13 0x000025b8b33dbe1d in ?? ()
#14 0x0000214054f03dc1 in ?? ()
#15 0x000025b8b33dbd81 in ?? ()
#16 0x00007ffdbf6ff590 in ?? ()
#17 0x0000000000000006 in ?? ()
#18 0x00007ffdbf6ff6a8 in ?? ()
#19 0x000025b8b66b6767 in ?? ()
#20 0x4032000000000000 in ?? ()
#21 0x402a000000000000 in ?? ()
#22 0x0000000000000000 in ?? ()

[Switching to thread 10 (Thread 0x7f4bd51e3700 (LWP 2002))]
#0  0x00007f4bd6de5b0f in futex_wake (private=<optimized out>, processes_to_wake=2147483647, futex_word=0x3b13d94)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:231
231     ../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
(gdb) bt
#0  0x00007f4bd6de5b0f in futex_wake (private=<optimized out>, processes_to_wake=2147483647, futex_word=0x3b13d94)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:231
#1  __condvar_dec_grefs (private=<optimized out>, g=1, cond=0x3b13d80) at pthread_cond_wait.c:160
#2  __pthread_cond_wait_common (abstime=0x0, mutex=0x3b13d58, cond=0x3b13d80) at pthread_cond_wait.c:559
#3  __pthread_cond_wait (cond=0x3b13d80, mutex=0x3b13d58) at pthread_cond_wait.c:655
#4  0x0000000000a53fe9 in uv_cond_wait (cond=<optimized out>, mutex=<optimized out>) at ../deps/uv/src/unix/thread.c:735
#5  0x0000000000962b64 in node::BackgroundRunner(void*) ()
#6  0x00007f4bd6ddf6db in start_thread (arg=0x7f4bd51e3700) at pthread_create.c:463
#7  0x00007f4bd6b0888f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[Switching to thread 11 (Thread 0x7f4bd49e2700 (LWP 2003))]
#0  __GI___libc_free (mem=0x7f4bc81275c0) at malloc.c:3101
3101    malloc.c: No such file or directory.
(gdb) bt
#0  __GI___libc_free (mem=0x7f4bc81275c0) at malloc.c:3101
#1  0x0000000000f15ab8 in void v8::internal::MemoryChunk::ReleaseSlotSet<(v8::internal::RememberedSetType)1>() ()
#2  0x0000000000efa3dc in v8::internal::RememberedSetUpdatingItem<v8::internal::MajorNonAtomicMarkingState>::Process() ()
#3  0x0000000000ee1afb in v8::internal::PointersUpdatingTask::RunInParallel() ()
#4  0x0000000000edf19f in v8::internal::ItemParallelJob::Task::RunInternal() ()
#5  0x0000000000bc65f6 in v8::internal::CancelableTask::Run() ()
#6  0x0000000000962bb9 in node::BackgroundRunner(void*) ()
#7  0x00007f4bd6ddf6db in start_thread (arg=0x7f4bd49e2700) at pthread_create.c:463
#8  0x00007f4bd6b0888f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Pretty neat. I wonder if there is anything I can change about my code or environment that would encourage this crash, or avoid it. Or maybe this is just something in v8, like the exit race thing you mentioned? I appreciate the help!