Parallel message processing

Parallel message processing

It is possible to configure your .NET Worker Service to consume messages from the topic partitions in parallel. This will increase message throughput.

Configuration

The number of parallel consumers is defined in the appsettings.json:

{
  "Kafka": {
    "ConsumerGroups": [{
       //
       "ParallelConsumers": xxx
       //
    }]
  }
}

Thread-safety

Each consumer will be accumulating its own intake of messages from the partitions assigned. Each consumer will be creating its own instance of controller, so you should not worry about thread-safety of your controllers as they won't be accessed by multiple threads in parallel.

However, if your controller instances use a shared dependency, this dependency may be called from multiple threads in parallel.

Number of parallel consumers

In most cases, number of parallel consumers = number of partitions, but not necessarily.

Please follow these general rules:

1. Number of parallel consumers must not be greater than the number of partitions

Each consumer tries to consume from at least 1 partition, so having more consumers than partitions will not increase your throughput. Excessive consumers will simply waste resources and won't be getting any messages.

Also, if you run multiple instances of your .NET Worker Service, the total sum of parallel consumers in each service shouldn't be greater than the number of partitions.

2. Number of partitions should be divisible by the number of parallel consumers

This comes from the fact that, in order to have the best throughput, you want to have even distribution of partitions among the consumers, i.e. each consumer has the same amount of partitions assigned.

Examples:

• OK: 10 partitions, 5 consumers => each consumer is assigned with 2 partitions
• OK: 10 partitions, 10 consumers => each consumer is assigned with 1 partition
• BAD: 10 partitions, 7 consumers => there are most likely 3 consumers with 2 partitions + 4 consumers with 1 partition, therefore some partitions might be consumed slower.

Partitions assignment

The partitions are assigned to your consumers automatically. This is done by Kafka.

Currently, the library does not support manual partition assignment. If you want this feature, please contribute or create a feature request.