Add Distributed Locking to Prevent Race Conditions in Pipeline Declaration #116
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When RoadRunner starts its worker pool for the
jobs
plugin, each worker process attempts to declare consumer pipelines during initialization. This creates a race condition where multiple workers try to create the same pipeline simultaneously, potentially leading to conflicts, duplicate creation attempts, and errors.This PR introduces a distributed locking mechanism to the
RPCPipelineRegistry
class to ensure that only one worker process can declare a specific pipeline at a time during the worker pool initialization phase.This issue is particularly problematic for the Kafka driver, where each pipeline declaration creates a Kafka consumer that joins the consumer group. When multiple workers declare the same pipeline simultaneously, phantom consumers are created in the Kafka consumer groups. These phantom consumers remain idle and don't consume messages from their assigned partitions, while only one consumer actually processes messages. This leads to: