Partition allocation and even distribution and resource usage across consumers in a group

[rajatgoyal247341]

Let's say I have topic with 15 partitions, which is currently consumed by a group of 3 consumers and each running on a difference instance.

Now with parallel consumers :

a) For achieving parallel partition order, is it always a good practice to start with good high max-concurrency ( even if some threads are idle). If I execute 20 max-concurrency on each instance, 5 threads will run on each consumer and rest will be idle. I am asking this because if we downscale our system ( remove an application node) from 3 to 2, will it be running 7 & 8 (total 15) threads per instance ? Does it auto-adjust the thread-pool size on every instance, based on no-of consumers in group ?

b) In same fashion (for achieving parallel partition order) , if we add more consumer ( add an instance) to the group, does load sharing among instances change automatically ? For example if I increase consumers from 3 to 4, will each instance be running 4,4,4,3 threads respectively,

c) For achieving unordered / key ordering : In case of upscale / downscale, does load distribution adjustment happens automatically among instances.

[astubbs]

Hi @rajatgoyal247341 , do you not want any help with this anymore?

The max concurrency setting is per instance only, not global. However, this could potentially be changed to an optionally global setting. Is that something that you'd find useful?