Way to cancel stuck running jobs

[topofstack]

Hi, is there any way to cancel stuck running jobs? Or we only can rely on background rescuer service.

[bgentry]

There's the JobCancel API on the client for doing this programmatically and it's exposed in the UI as well. However be aware that this is only a request to cancel the job—ultimately if the job doesn't respond to context cancellation then it won't stop running (there's no mechanism to forcefully kill a goroutine).

[brandur]

Yep, +1. And to be clear, we generally use the term "stuck" to refer to jobs that are not responding to context cancellation. @topofstack Just out of curiosity, are you finding you're running into stuck jobs a lot in the real world? I ask because we've been trying to implement a few features recently to help compensate for these.

[topofstack]

Thanks

I have long-running jobs (10–20 min) with a dynamic timeout defined by the user. If job logic gets stuck in a long blocking operation that can't be cancelled gracefully, and the worker dies, there is no way to cancel it manually. I then have to wait for the rescue timeout, which can be quite long.
Not that it happens often; my case is probably pretty niche

[bgentry]

@topofstack yeah, the setup where your jobs enter a long blocking operation that does not respond to context cancellation is one we have few options for in Go. It's possible to more aggressively detect the "stuck" job and allow it to be restarted elsewhere, or you can kill the Go process—that's really it. This is why we recommend that all jobs respond to context cancellation any time there's a potentially long operation.

[topofstack]

But what if I set job status manually by executing update river_job set state = 'discarded' where id = $1. Will it break anything?

[brandur]

Setting the job to discarded manually shouldn't cause too many issues. The only things to watch out for:

• In case the job ever were to succeed, it will not be set to completed/errored because the executor will only do so if the job was still in running state. So a discarded job will stay discarded.
• You may still end up starving out your worker pool. Every job that gets stuck will eat a goroutine, so if you have too many of these during one session you may eventually end up starving out the whole pool.