Worker underutilization due to delayed queue publishing

[ctrochalakis]

Problem:

In rspecq, the master worker is responsible for preparing the job queue and publishing it to Redis. This process involves identifying all spec files, splitting slow files into individual examples (if file_split_threshold is set), and ordering them based on their timings.

Currently, the master worker prepares the entire job list before publishing it to the queue. This can introduce a significant delay, especially in large test suites with many slow files that need to be split. During this time, all other workers remain idle, waiting for the queue to become ready. This leads to under-utilization of worker resources and increases the overall test execution time. In our suite, we measured that time to be around 90 seconds.

Proposed Solution:

To address this issue, we propose introducing a mechanism to "early push" a batch of jobs to the queue, before hitting the time-consuming file-splitting procedure. This will allow other workers to start processing jobs while the master continues to prepare the rest of the queue.

The proposed solution involves the following changes:

Modify the queue publishing logic to allow pushing jobs in multiple batches without immediately marking the queue as ready.
Update the master worker's logic to:
a. First, push a batch of non slow-running spec files to the queue.
b. Then, process the slow-running spec files (splitting them into individual examples).
c. Finally, push the remaining jobs to the queue and mark it as ready.
Introduce a new option, --early-push-max-jobs options, that controls the maximum number of jobs to be pushed early. You can set this to your number of deployed workers.

Note that this feature has an effect on the queue's scheduling fairness, since it might push faster jobs, before slower ones that are splited from a larger spec file. We mitigate that with the --early-push-max-jobs. By tweaking this value you can control the number of early pushed jobs, thus reduce the unfairness of the queue. We plan to set this to the number of our workers, to make sure everyone of them gets utilized.