Fixed a locking race condition

[mark-rushakoff]

We found a race condition where two workers would occasionally both complete the same job. The first worker to finish the job finishes successfully, but the second worker would try to delete the job (thinking it finished successfully) and that worker process would crash as a result.

This pull request addresses this race condition.

[JonathanTron]

Hi @mark-rushakoff, thanks a lot for this pull-request!

I've checked the reason of the spec failure and despite your changes look great it does not cover the full delayed_job's expectations about Delayed::Job.reserve. For instance a job can be reserved again even when reserved by another worker if it took longer than the Delayed::Worker.max_run_time.

I think the problem lies somewhere else, I will push a change I made thanks to your PR, it would be great if you can test it and tell me if it fixes your problem.

[cf-frameworks]

We are still seeing errors with your newest code: Please see PR: #3 for tests that show the error.

[JonathanTron]

Thanks a lot for the test case, I will check it out.

Out of curiosity what db are you using?

[mark-rushakoff]

We've been testing with (and seeing issues in production with) MySQL.