Parallelism and Concurrency

[andrew-sol]

Here is a section from the doc:

Secondly, you can run as many workers as you want. Every worker will run in parallel if it has a CPU at its disposal. You can run several workers in a given machine if the machine has more than one core, but you can also run workers in totally different machines. The jobs running on different workers will be running in parallel, so even if the job is CPU-intensive you will be able to increase the throughput which will normally scale linearly with the number of workers.

It’s not clear how it works. Consider the following setup:

const queue = new Queue('main');
const jobHandler = async (job) => {};

const worker1 = new Worker(‘main’, jobHandler);
const worker2 = new Worker(‘main’, jobHandler);
const worker3 = new Worker(‘main’, jobHandler);
const worker4 = new Worker(‘main’, jobHandler);

Question 1: Will it occupy 4 CPUs under high load or will it work the same way as a single worker instance with the concurrency = 4 and use only 1 CPU?
Question 2: If I run that setup on many separate machines connected to the same Redis, is it guaranteed that the same job won’t be picked up by two of them at the same time and processed twice (or more times) simultaneously?

[manast]

Question 1: Will it occupy 4 CPUs under high load or will it work the same way as a single worker instance with the concurrency = 4 and use only 1 CPU?

As NodeJS is single-threaded, if you instantiate several workers in the same NodeJS process then they will not use more than 1 CPU in total. You can use PM2 for example to start different NodeJS processes that will effectively run in parallel if you have enough CPU cores.

Question 2: If I run that setup on many separate machines connected to the same Redis, is it guaranteed that the same job won’t be picked up by two of them at the same time and processed twice (or more times) simultaneously?

BullMQ guarantees at least once semantics, and in general, you will get exactly once semantics independent of the number of workers, however, if your worker stalls due to heavy CPU usage for more than 30 seconds (by default), then the job will be considered stalled and moved back to "wait" status so that other worker will start working on it. If the first worker still is able to continue working on the job, then the same job will be processed twice. If this risk is not acceptable for you, the solution is to use idempotent jobs.

[andrew-sol]

Thanks, that's pretty clear. This sentence from the doc is quite confusing though:

Every worker will run in parallel if it has a CPU at its disposal.

I thought the workers could spawn child processes automatically.

[manast]

Btw, it would be possible to achieve "at most once" semantics by setting the "maxStalledCount" option to 0 in the workers, since that will just mark as failed jobs when the worker is not able to renew the job locks in time: https://api.docs.bullmq.io/interfaces/v5.WorkerOptions.html#maxStalledCount

[manast]

(by default this setting is 1, so in practice, this means that by default the queues are "at least once" and "at most twice" semantics)