Keep order of jobs in group upon retry

[freiondrej-lokalise]

I am using group mode with per-group concurrency = 1. Let's say I add three jobs in it, job A, job B and job C, to be executed as A-B-C. Job A is being executed first. However, when it fails and is being retried, the order changes to B-C-A. I'd like the per-group order to be kept as A-B-C and only if A exhausts its attempts, it would be discarded, so only B-C would be executed.

I tried to use per-group priority for this, but didn't find a reliable way of setting the priority accordingly to my needs. What I tried was to use queue.getGroupJobsCount() when adding to the queue and incrementing it by 1, but it seems it's not fully reliable as sometimes the count is not up to date, so I end up with two identical priorities which then does not work as I'd need.

Maybe a new option would be needed for this, something like workerOptions: {group: {concurrency: 1, keepOrderOnRetry: true}}?

[roggervalf]

hi @freiondrej-lokalise, we only support priorities intra groups, it may be possible to use priorities for groups but it may imply a breaking change. We are in the building the new breaking change for bullmq that also will affect pro version, in that new breaking change could be included

[freiondrej-lokalise]

Thanks for your comment @roggervalf ! 🙌

we only support priorities intra groups, it may be possible to use priorities for groups but it may imply a breaking change

I'm not suggesting to use priorities for groups, only to allow configuring how prioritization intra groups is handled. Let me know if I should elaborate or illustrate with a better example 🙏

[roggervalf]

hey, I misunderstood you. In reality I worked on some ideas for this case #2465 or #2687, we are still evaluating which is the best way to handle it. Also I have one extra approach but probably we will revisit this after our next breaking change that we are currently working