Skip to content

Conversation

butschster
Copy link
Member

When RoadRunner starts its worker pool for the jobs plugin, each worker process attempts to declare consumer pipelines during initialization. This creates a race condition where multiple workers try to create the same pipeline simultaneously, potentially leading to conflicts, duplicate creation attempts, and errors.

This PR introduces a distributed locking mechanism to the RPCPipelineRegistry class to ensure that only one worker process can declare a specific pipeline at a time during the worker pool initialization phase.

This issue is particularly problematic for the Kafka driver, where each pipeline declaration creates a Kafka consumer that joins the consumer group. When multiple workers declare the same pipeline simultaneously, phantom consumers are created in the Kafka consumer groups. These phantom consumers remain idle and don't consume messages from their assigned partitions, while only one consumer actually processes messages. This leads to:

  • Unprocessed messages sitting in partitions assigned to phantom consumers
  • Reduced throughput as available partitions are not being consumed

@butschster butschster added this to the v3.x milestone Jun 5, 2025
@butschster butschster requested a review from roxblnfk June 5, 2025 13:41
@butschster butschster added bug Something isn't working Component: Queue labels Jun 5, 2025
@roxblnfk roxblnfk merged commit 7d8c704 into 3.x Jun 5, 2025
4 of 5 checks passed
@roxblnfk roxblnfk deleted the hotfix/pipeline-declaration branch June 5, 2025 14:33
@roxblnfk roxblnfk mentioned this pull request Jun 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Component: Queue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants