-
Notifications
You must be signed in to change notification settings - Fork 25
Open
Description
We have a cron-like task that scans a large Redis dataset and takes ~35 minutes to complete.
With RedisStreamBroker
, the default idle_timeout
is 600_000 ms (10 minutes).
Once the worker runs longer than that, TaskIQ calls XAUTOCLAIM
, redelivers the message, and a second instance starts (ours cancels itself via a custom lock, but it still creates noise).
Right now, the only workarounds we see are:
- Split the long task into shorter pieces.
- Set
idle_timeout
to a huge value (we currently use one year in milliseconds).
Questions:
- Is there any supported way to completely disable automatic redelivery, so a job is never re-queued while its worker is alive?
- If disabling isn’t possible, what’s the recommended approach for genuinely long-running jobs?
- Was the fixed redelivery behavior chosen mainly for resilience against crashed workers, or do you expect users to handle long tasks differently?
Thanks!
smalyu and bashirmindee
Metadata
Metadata
Assignees
Labels
No labels