PM2 does not restart crashed processes #5396
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With
autorestart
set to true, PM2 will restart child processes if they disconnect. However, if the process crashes internally, PM2 will not attempt to restart or replace it, leaving the overall service running with one less process.In our case, we just upgraded to node 16 and PM2 5.2.0, and are seeing node periodically throw
ERR_IPC_CLOSED
now. We're not sure about the root cause of that error yet, but regardless, we would expect PM2 to either handle that or at least restart the crashed process. Unfortunately, it's doing neither and just letting the process permanently die.This appears to be because PM2 marks the process as
ERRORED_STATUS
in itserror
event handler, and ERRORED_STATUS processes do not even attempt to restart.My proposed fix here skips marking the process as ERRORED_STATUS on a single
error
event, instead reserving that status for repeated crashes. This is the simplest solution, but I'm open to other ideas/approaches here.