Settings

Settings

Update your appsettings.json to include new Kafka section.

Here is the minimum settings that you should provide for a consumer group:

{
  "Kafka": {
    "ConnectionString": "xxx",
    "ConsumerGroups": [{
       "GroupId": "xxx",
       "TopicName": "xxx",
       "ParallelConsumers": xxx,
       "Intake": {
         "Strategy": {
           xxx
         }
       }
    }]
  }
}

ConnectionString

Your Kafka connection string, e.g. broker-1:9092,broker-2:9092,broker-3:9092. You can omit this setting in the appsettings.json if you provide it using environment variables or using another configuration provider.

ConsumerGroups

Array of consumer group configurations. See also Multiple consumer groups.

GroupId

Name of your Kafka consumer group. This name should be unique for each group of consumers that consumes messages from the same topic. For example, if you have a topic "A" and you have several independent groups of consumers that should receive messages from this topic, then each group must have its unique name (GroupId).

TopicName

Name of the topic that you want to consume messages from.

ParallelConsumers

Number of parallel consumers that will be subscribed to the kafka topic. Notes:

• this value shouldn't be greater than the number of partitions in the topic. Each consumer tries to consume from at least 1 partition, so having more consumers than partitions will not increase your throughput.
• if you run several instances of your .NET Worker Service, make sure that the sum of ParallelConsumers in each of these instances are not greater than the number of partitions in the topic.
• the minimum value is 1 so that you must have at least 1 consumer.

Intake

This section defines the behavior for each intake of messages.

Strategy

Specify one of the 3 default intake strategies that suits your purposes or omit this setting if you provide your custom intake strategy (see more in Intake and intake strategies):

FixedSize

Process every X messages:

"Strategy": {
  "Name": "FixedSize",
  "Size": X
}

FixedInterval

Process messages every Y milliseconds:

"Strategy": {
  "Name": "FixedInterval",
  "IntervalInMs": Y
}

MaxSizeWithTimeout

Process either every X messages or every Y milliseconds (whichever happens first):

"Strategy": {
  "Name": "MaxSizeWithTimeout",
  "MaxSize": X,
  "TimeoutInMs": Y
}

Other settings

There are additional settings that you can provide to configure error handling, dead-lettering, throttling, or streaming. Please visit the corresponding articles where it is explained.