Possible race condition when completing jobs

[josenavas]

If a job has more than one validator, sometimes the validators are set to "waiting" but the job never finish (i.e. deadlock). My guess is that the race condition is happening in this call. If the validators are executing that call at the same time, they will not see that the other job is already in the waiting status because the entire operation is executed inside a transaction, so both jobs end up waiting for the other job to complete.

Possible solutions that I can think of:

1. Instead of performing the counting by checking how many jobs are not in "waiting", put a specific counter with a lock in which serialization of operations is ensured by using a "ROW EXCLUSIVE)" lock.
2. Explicitly commit the change of status before performing that check. The amount of code changes may be higher than it may seem as there are multiple nested contexts that creates transactions, and enforcing a commit may break some other unexpected functionality.

I think (1) is the more correct and easy solution although it requires DB changes.

[antgonza]

Looking at the code and behavior perhaps an easier and more accurate change will be to not finish the main job and just put the .release_validators() code at the end in a while look until is done with a sleep(10)?

Just to be clear, currently if we are doing operation X and it has two validators Y/Z, we are creating 1 job x, at the end (before it finishes) it starts the job validators y/z and the deadlock happens cause both of the y/z finish at the same time and don't finish the actual process cause each other haven't seen the other job finished.

[antgonza]

closed via #2203