Node v0.11.16 doesn't exit on SIGTERM when runs inside Docker

[skozin]

Dockerfile:

FROM node:0.11.16
WORKDIR /app
COPY index.js /app/
CMD ["node", "index.js"]

index.js:

console.log('started');
process.stdin.resume();

This app will ignore TERM and INT signals, so it cannot be terminated using Ctrl+C or docker stop. Tested with Docker 1.4.1, Linux kernel 3.16.7.

Other configurations:

• Node.js v0.11.16 in Docker, built from source (Alpine Linux, musl-libc): doesn't exit;
• Node.js v0.11.16 in Mac OS X, outside of Docker: correctly exits;
• Node.js v0.10.36, both in Docker and on Mac OS X: correctly exits.

It appears that Node actually recieves and handles these signals, but for some reason doesn't perform the default action. This can be verified by adding signal listeners in index.js:

console.log('started');

exitOnSignal('SIGINT');
exitOnSignal('SIGTERM');
process.stdin.resume();

function exitOnSignal(signal) {
  process.on(signal, function() {
    console.log('\ncaught ' + signal + ', exiting');
    process.exit(1);
  });
}

It correctly prints Caught SIG{INT,TERM} and exits.

Initially I posted this to docker-node repo, but moved it here as it seems unrelated to the particular Docker image.

[misterdjules]

The change in behavior was introduced by c61b0e9. With this commit, when a user sends SIGINT by pressing CTRL-C in a terminal where node is the foreground process, the node process doesn't exit explicitly and instead forward the signal.

The reason why this change has an impact on how a node process behaves within a node container is that, when using the "exec" form of the Dockerfile CMD instruction like following:

CMD ["node", "index.js"]

the node process has the pid 1, and thus cannot be killed by SIGINT or SIGTERM. So forwarding the signal to itself is not sufficient for node to exit.

However, if one uses the "shell" form of the Dockerfile CMD instruction like following:

CMD node index.js

then the pid of the node process within the container is not 1, and forwarding the SIGINT signal works as expected, that is the process exits properly.

I believe the correct issue to track in the docker repository is moby/moby#2436.

Closing as it seems to be an issue with docker with an easy workaround, but please feel free to continue the discussion if this solution does not fix your issue.

Thank you!

[skozin]

@misterdjules, thank you for explanation! I was unaware of this feature of PID 1, but now it seems quite logical :)

Unfortunately, the "shell" form of CMD is not a workaround, because sh doesn't pass signals to its children. Ctrl+C works because terminals send SIGINT to all processes in a group, so node process recieves it directly from the terminal.

But docker stop and docker kill commands send signals to PID 1, and Node.js doesn't recieve them and cannot exit cleanly; it gets killed by Docker with SIGKILL after a timeot. Here is an example:

process.on('SIGTERM', function() {
  console.log('\ncaught SIGTERM, stopping gracefully');
  process.exit(1);
});
console.log('lauched'); process.stdin.resume();

Dockerfile:

FROM node:0.11.16
COPY index.js /app/
CMD node /app/index.js

Trying to stop:

# (terminal 1)
$ docker build -t 'test' . && docker run --rm -it --name 'test-app' 'test'
launched

# (terminal 2)
$ docker exec 'test-app' ps
PID   USER     TIME   COMMAND
    1 root       0:00 /bin/sh -c node index.js
    7 root       0:00 node index.js
   12 root       0:00 ps

$ docker kill -s TERM 'test-app'
# the app doesn't recieve the signal and continues to run
$ docker kill -s INT 'test-app'
# same here
$ docker exec 'test-app' kill -s INT 1
$ docker exec 'test-app' kill -s TERM 1
# no reaction
$ docker stop 'test-app'
# the app doesn't recieve SIGTERM and continues to run, and gets killed
# with SIGKILL after a timeout; nothing is printed in terminal 1

The only easy fix I see here is to use the "exec" form of CMD, handle signals by hand and call process.exit(), emulating the previous behavior of Node. Another option is to write a wrapper script that will launch Node and forward all signals, but it is not easy to do this correctly.

Now it's clear to me that this is a Docker issue, and that Docker should run a special process as each container's PID 1, which will launch the main process and handle signal forwarding, child reaping, etc.

[misterdjules]

Thank you @skozin for your great writeup about this issue. I'm sure other users of docker and Node.js will find it very useful!

[DJviolin]

My solution for the PID1 problem:

The Dockerfile ends with:

ENTRYPOINT ["/bin/bash", "-c"]

Than I run with this shell script, which has the filename of node and have chmod +x:

#!/bin/bash

docker run --rm -it -p 8083:80 -v $HOME/node/work/:/root/node/:rw node_node "echo pid1 > /dev/null && node $@"

The trick is "echo pid1 > /dev/null && node $@" which is the command. $@ is a shell script to accept user input from command line for example: ./node -v which will return the version of Node.js inside the running container.

So echo will trap PID1 and sending output to /dev/null.

Here is my Dockerised Node.js development environment: https://github.com/DJviolin/Node.js-Development-Environment