Allow duplicate jobs to be discarded

[mhenrixon]

Should resolve #176

I want to prevent the following:

These jobs can under no circumstances end up in this situation, discarding the new jobs instead of queuing them ensures this won't happen.

I took the simplest possible approach using terminology from sidekiq-unique-jobs. All feedback appreciated.

[rosa]

Hey @mhenrixon, thanks for this! There was already a PR for it: #523
But this idea of marking the job as finished is really clever! It'd allow for quite a few simplifications 🤔 I wonder if it's too clever, though.... would it lead to confusing scenarios where you don't know what happened to a job? Although I guess the same is true if the job is simply discarded, so this should be ok!

I need to review what happens here for the bulk enqueue case, which was the tricky part of that other PR.

[mhenrixon]

Hey @mhenrixon, thanks for this! There was already a PR for it: #523

I need to learn to read all the issues. Sorry about the extra PR.

But this idea of marking the job as finished is really clever! It'd allow for quite a few simplifications 🤔 I wonder if it's too clever, though.... would it lead to confusing scenarios where you don't know what happened to a job? Although I guess the same is true if the job is simply discarded, so this should be ok!

After working on sidekiq-unique-jobs, and with sidekiq. I have learned that some visibility is appreciated. One of the reasons to use a database-backed queue system is to maintain visibility and avoid paying a blackmailer for keeping the data safe.

I am happy to close this if you prefer the other one @rosa. I didn't look into the other PR in detail, but the bulk enqueue sounds important.

[rosa]

Oh, no worries! I think the idea here is better than what we were trying to do in the other PR, so I think combining these two would be the best! I need to review the other one again after some changes so I'll review these two together and will think how to proceed, maybe transplanting some commits over 🤔

[mhenrixon]

Feel free to add commits or let me know what to pick or how I can help land this in the main branch. I am at your service!