Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pause, Resume, Kill/Cancel a Running Job #2650

Closed
JamesLAllen opened this issue Jan 27, 2024 · 3 comments
Closed

Pause, Resume, Kill/Cancel a Running Job #2650

JamesLAllen opened this issue Jan 27, 2024 · 3 comments

Comments

@JamesLAllen
Copy link

JamesLAllen commented Jan 27, 2024

Is your feature request related to a problem? Please describe.
I would like the ability to pause, resume, and kill/cancel a job that is currently running and events that we can listen for our own custom jobs.

Describe the solution you'd like
Currently when I try to cancel a running job, like a full re-index which takes a very long time, it gives me an error saying I can't stop a job while it's running. I would like the ability to pause a job and then resume it, or cancel it altogether.

Describe alternatives you've considered
I'm not sure what alternatives there would be.

@michaelbromley
Copy link
Member

Agreed, the inability to kill a running job can be a pain. especially when they occasionally get into a false "running" state and there's no easy way to evict it from the queue.

I would like the ability to pause a job and then resume it

This sounds like it might be rather complex to get right, since once the process function begins execution, I don't know of a general way to "pause" that execution. I don't know of any prior art on a job queue system supporting this kind of functionality. Do you?

@JamesLAllen
Copy link
Author

I need to know more about the events within the underlying job queue system, specifically how do you pass along percentage updates?

I might be wrong here but it doesn't look like we assign a unique ID to each job, so there's currently no way to modify it once it's been kicked off, right? I think this is how I might tackle this specific improvement:

  1. Assign a unique ID for all jobs.
  2. Assign a unique ID for the worker that accepts a job.
  3. Add job status columns or a table and expose hooks for jobs that are created to update, modify, cancel and delete.
  4. Worker is responsible to check back periodically for changes.
  5. In addition, Server would also check as it does on running jobs to ensure nothing has timed out or errored. If the job hasn't responded in a while, then Server assumes the job has stalled and releases the worker, freeing it up to be taken by another worker or cancelled depending on rules that are set by the admin (some jobs should be restarted infinitely, others should be cancelled, others only restarted a set number of times).

@michaelbromley
Copy link
Member

Related:

michaelbromley added a commit that referenced this issue Feb 2, 2024
Relates to #1127, relates to #2650. This commit adds a mechanism to
track cancellation of jobs in Redis using pub/sub on a custom
set which tracks the IDs which have been cancelled.

The job process functions will still continue to execute unless
there is specific logic to check the job state and throw an
error on cancellation.
michaelbromley added a commit that referenced this issue Feb 2, 2024
Relates to #1127, relates to #2650. This commit alters slightly the way cancellation
is handled: when a job is cancelled, it's status is updated but the
job will continue to run unless the `process` function explicitly
throws an error.
michaelbromley added a commit that referenced this issue Feb 2, 2024
Relates to #1127, relates to #2650. This commit handles the case of a job
being cancelled for those built-in jobs that can be long-running.
michaelbromley added a commit that referenced this issue Feb 2, 2024
@michaelbromley michaelbromley moved this from 📅 Planned to 🏗 In progress in Vendure OS Roadmap Feb 2, 2024
@michaelbromley michaelbromley moved this from 🏗 In progress to 🔖 Ready in Vendure OS Roadmap Feb 9, 2024
@michaelbromley michaelbromley moved this from 🔖 Ready to ✅ Done in Vendure OS Roadmap Apr 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: 🚀 Shipped
Development

No branches or pull requests

2 participants