`SolidQueue::Processes::ProcessPrunedError` shown in "Failed jobs" and not being released back to their queues

[salmonsteak1]

Hey there, the README states this :

If processes have no chance of cleaning up before exiting (e.g. if someone pulls a cable somewhere), in-flight jobs might remain claimed by the processes executing them. Processes send heartbeats, and the supervisor checks and prunes processes with expired heartbeats, which will release any claimed jobs back to their queues. You can configure both the frequency of heartbeats and the threshold to consider a process dead. See the section below for this.

From what I understand, this means that when the plug is pulled or the worker is terminated unexpectedly, jobs that are still in the midst of processing by that worker will be released back to their queue the next time the supervisor checks and prunes processes with expired heartbeats.

However, it seems like I'm getting these ProcessPrunedError:

I believe this happens in the event of the "plug being pulled" (or something similar like forcefully shutting down a kubernetes pod), the supervisor will also be killed and the job will be stuck with the worker. I'm just not sure when/where this ProcessPrunedError is being generated, and if there's anything we can do about it for it to be re-enqueued

[wflanagan]

I've run into this as well. We implemented a query to find them, and then try to requeue them. But, it seems we've not done this well, and it's turned into a bit of a mess. It seems like when the process is terminated, we should throw them back to the queue to attempt again.

[rosa]

@salmonsteak1, oh yes! I forgot to update the README on #277. In-flight jobs whose worker terminates abnormally will now be marked as failed, just like if the job failed, so you can inspect them and retry/discard/fix them manually.

The reason is that the worker might be crashing because of the job. For example, a job that has a memory leak causes some monitoring process to kill a worker because of memory consumption. If we simply put the job back into the queue when the process is killed, another worker will pick it up just to be killed and so on. I need to update the README.

@wflanagan you can use Mission Control to inspect these jobs and retry them manually. If you are getting these often/regularly, something must be up with your setup, as it should be an exceptional situation. Regular deploys, if they're implemented with a small shutdown time where workers have time to terminate orderly would not cause this, as in-flight jobs whose workers terminate orderly but didn't have time to terminate them will be put back into the queue.

[wflanagan]

@rosa Hey Rosa, thanks.

But, I will say that this is still the same error that we originally discussed and is an open ticket, namely that a worker that fails leaves abandoned jobs.

My setup (Kubernetes and workers that might get killed) is a pretty stock setup. Since this happens, I've been trying to go find the ones that have errored in this way, and requeue them. That's the only automation that we've added to this.

Given that we process 500k jobs/day, and our workers cycle 3 to 10 times a day (CI deployments, worker errors, etc), and we have longer running jobs (a few minutes at times), this unfortunately is a common situation that our workers will eventually get ungracefully shutdown.

The jobs being thrown to failed isn't ideal. And, recovering them starts making things throw other errors.

[rosa]

@wflanagan, something must be up still, as workers shouldn't be terminated in a way that jobs are failed. A deploy that sends a SIGTERM to the workers followed by a SIGQUIT wouldn't cause jobs to fail, they'd be released to their queues. What would cause this is for example a SIGKILL, a cable being plug, a server dying completely... I think you need to investigate why this is happening so often in your case. For comparison, we run ~20M jobs a day (some of them long-running) and deploy many times every day (some days 3 times, some days 15 times... it depends), and the last time we saw a worker terminating ungracefully was about a month ago, when we had a memory leak in our of our jobs, and the worker was being killed (SIGKILL) due to that. It was also consistent: retrying the job would kill the worker again. We fixed the memory leak (something related to our application logic) and that was it.

[rosa]

I just updated the README with accurate information after the changes introduced in #277. Thanks for spotting this, @salmonsteak1 🙏