addBulk causes delay in job pickup with multiple worker processes

[skirsten]

Version: Python v2.9.3

Given this very simple process function:

async def process(job: Job, job_token: str):
  print(datetime.datetime.now().isoformat(), "process job", job.name)

  await asyncio.sleep(10)

and the sample worker from https://docs.bullmq.io/python/introduction#get-started running as three separate processes on the same node to the same local redis.

If I then insert jobs using addBulk (from python or node):

await queue.addBulk([dict(name=f"test-{i}") for i in range(3)])

there is a very large delay between start of the jobs:

worker-0 2024-09-03T11:15:31.301039 process job test-0
worker-1 2024-09-03T11:15:32.809889 process job test-1
worker-2 2024-09-03T11:15:33.839579 process job test-2

as can be seen in this example there is at least a second between the workers processing the jobs. Most times its even more and the delay scales with the number of workers.

This does NOT happen when using add in a loop:

for i in range(3):
  await queue.add(f"test-{i}", {})

or if using "concurrency": 3 in a single worker:

worker-0 2024-09-03T11:14:34.423502 process job test-0
worker-1 2024-09-03T11:14:34.460346 process job test-1
worker-2 2024-09-03T11:14:34.500095 process job test-2

For my use-case I need to add a flow with a lot of children which behaves like addBulk. This causes it to not scale well to the huge number of workers I need. So unfortunately the loop workaround doesn't work for the flow.