Throttling

Throttling

The library allows you to limit the rate at what messages are being consumed and sent to your controller.

By default, a Kafka consumer tries to consume as many messages as possible.

However, if during the message processing you make requests to a database, external API, etc. you may want to avoid putting them at excessive load in case of:

• message spikes (when suddenly too many messages are published to the topic),
• cold start (when there are many messages from the previous days that you want to gradually process and catch up with the current flow of messages),
• service pause (when your service was paused and many messages have arrived into the topic since).

Configuration:

Specify MaxSpeed value for your consumer group configuration in the appsettings.json:

{
  "Kafka": {
    "ConsumerGroups": [{
       "GroupId": "<group-1>",
       "Intake": {
         "MaxSpeed": xxx // <-- supply your value
       }
    }]
  }
}

This value means the maximum amount of messages to be handled for 1 partition per 1 second.

Examples:

• if you have 1 partition and MaxSpeed is 500, you will be handling no more than 500 messages each second,
• if you have 10 partitions and MaxSpeed is 500, you will be handling no more than 5000 messages each second in total (10*500).

This setting is optional and you can omit it to allow unlimited rate.