Multiple consumer groups

Multiple consumer groups

The library allows you to run as many consumer groups as you wish.

Different consumer groups can consume either from different topics or from the same topic depending on your use cases.

Configuration

You can add multiple consumer groups in the appsettings.json:

{
  "Kafka": {
    "ConsumerGroups": [{
       "GroupId": "<group-1>"
    },
    {
       "GroupId": "<group-2>"
    }]
  }
}

You should then configure their options (Program.cs):

Host
    .CreateDefaultBuilder(args)
    .ConfigureServices((ctx, services) =>
    {
        services.AddKafkaEventLoop(ctx.Configuration, o => o
            .HasConsumerGroup("<group-1>", cgOptions => cgOptions
                .HasMessageType<MyMessage>()
                .HasJsonMessageDeserializer()
                .HasController<MyController>()
                .Build())
            .HasConsumerGroup("<group-2>", cgOptions => cgOptions
                .HasMessageType<MyOtherMessage>()
                .HasJsonMessageDeserializer()
                .HasController<MyOtherController>()
                .Build())
            .Build());
    })
    .Build()
    .Run();

Different connection strings

In most cases, your consumer groups exist in the same Kafka cluster.

However, it is possible to have consumer groups connected to different Kafka clusters.

In the following example, group-1 and group-2 use the global connection string aaa but there is a separate consumer group group-3 that overrides the global connection string and provides its own connection string bbb:

{
  "Kafka": {
    "ConnectionString": "aaa",
    "ConsumerGroups": [{
       "GroupId": "<group-1>"
    },
    {
       "GroupId": "<group-2>"
    },
    {
       "GroupId": "<group-3>",
       "ConnectionString": "bbb"
    }]
  }
}