If you have this problem, you should upgrade docker! [was: node aborts and core dumps as soon as it starts]

[bluesmoon]

Version

v18.1.0

Platform

Linux 4d7a71cb7acd 3.10.0-1160.62.1.el7.x86_64 #1 SMP Tue Apr 5 16:57:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Subsystem

No response

What steps will reproduce the bug?

start node from command line. no parameters

How often does it reproduce? Is there a required condition?

running inside a docker container, happens everytime when the container is on a CentOS host, but never happens on a MacOS host.

What is the expected behavior?

node should start

What do you see instead?

$ node
node[9]: ../src/node_platform.cc:61:std::unique_ptr<long unsigned int> node::WorkerThreadsTaskRunner::DelayedTaskScheduler::Start(): Assertion `(0) == (uv_thread_create(t.get(), start_thread, this))' failed.
 1: 0xb57f90 node::Abort() [node]
 2: 0xb5800e  [node]
 3: 0xbc915e  [node]
 4: 0xbc9230 node::NodePlatform::NodePlatform(int, v8::TracingController*, v8::PageAllocator*) [node]
 5: 0xb1b3d1 node::InitializeOncePerProcess(int, char**, node::InitializationSettingsFlags, node::ProcessFlags::Flags) [node]
 6: 0xb1bc89 node::Start(int, char**) [node]
 7: 0x7f2ca389fd90  [/lib/x86_64-linux-gnu/libc.so.6]
 8: 0x7f2ca389fe40 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
 9: 0xa93f0e _start [node]
Aborted (core dumped)

Additional information

No response

[bnoordhuis]

Looks like the RLIMIT_NPROC resource limit (ulimit -u) is set too low.

edit: or there's a seccomp filter active that blocks system calls node needs, like clone2/clone3.

[bluesmoon]

It's set to 1048576. In digging further, this seems to be limited to an Ubuntu 22.04 docker container running on a CentOS host. It works with other versions of Ubuntu and other host OSen

[bluesmoon]

I believe it's the same issue as documented here: https://askubuntu.com/questions/1405417/20-04-vs-22-04-inside-docker-with-a-16-04-host-thread-start-failures but I've been unable to find any solutions documented.

[bluesmoon]

Ok, I've isolated the problem to the docker version running on CentOS. After moving to docker-ce, the problem goes away.

Closing this ticket now.

[Dectom]

As a note, I have just experienced this issue, Any server I run with Docker version 20.10.7, build f0df350 has this issue however the ones that have Docker version 20.10.6, build 370c289 Do not.

I have just adjusted my image to use ubuntu 20.04 instead of 22.04

A Docker update might resolve this, However that's too destructive of an action and I can't update all servers to use new Docker Version at this time

[bluesmoon]

@Dectom it took about 5 minutes to go through the docker version upgrade. IAC, the problem is not with docker or with node, it's with CentOS.

[jaspertandy]

Just in case anyone finds this/finds it useful, I've just had almost the identical issue on a Gitlab runner whose host OS is Debian Bullseye, but the target image for a docker run command was on node:18-bookworm-slim (was actually node:18-slim but the underlying OS changed since bookworm was released) and locking to node:18-bullseye-slim fixed the issue. Will upgrade our runner and try again soon, but as a sort term fix this thread was very helpful in reaching this conclusion.

[Dectom]

@Dectom it took about 5 minutes to go through the docker version upgrade. IAC, the problem is not with docker or with node, it's with CentOS.

The systems we had this issue on were running Rocky Linux, whilst similar to CentOS it is different.

Also, for us a Docker version changes aren't really a plausible scenario, but the increase in version is what is the problem, any node on a later version has the problem vs the earlier version.

We reverted down to Ubuntu 20.04 instead of 22.04 as we can't upgrade docker across the nodes with the problem to see what later version resolves the problem.

I have ran into this same issue with the same docker version working and not working across other OS's we ran Debian 10 and Ubuntu host nodes and experience the same problem.

[Michiel-s]

Just in case anyone finds this/finds it useful, I've just had almost the identical issue on a Gitlab runner whose host OS is Debian Bullseye, but the target image for a docker run command was on node:18-bookworm-slim (was actually node:18-slim but the underlying OS changed since bookworm was released) and locking to node:18-bullseye-slim fixed the issue. Will upgrade our runner and try again soon, but as a sort term fix this thread was very helpful in reaching this conclusion.

Thank you @jaspertandy, you saved my day. I ran into the same issue and couldn't find the beginning of a solution.

Making explicit to use image node:18-bullseye instead of node:18 (which defaults to node:18-bookworm worked for me.