allow "brokers" parameter in client configuration to be function

[pimpelsang]

So instead of current:

const kafka = new Kafka({
  brokers: ['kafka1:9092', 'kafka2:9092']
})

Also allow passing callback

const kafka = new Kafka({
  brokers: async () => {
    // my service discovery logic here
    return ['kafkaIp1:9092', 'kafkaIp2:9092']
  }
})

When the connection to kafka is down, every reconnect attemt would also run rediscover and maybe get fresh IPs.

Another question: Right now the broker IP's can't be changed after client has been initialized?

[Nevon]

Right now the broker IP's can't be changed after client has been initialized?

Broker discovery is built into the Kafka protocol. We just need a connection to a single broker to be able to discover the rest. So the list of brokers that you specify when you create the client is only used for this initial discovery (we call them "seed brokers").

There is one scenario I could see is if all the brokers in the cluster are replaced while the consumer is connected, and you're not using a load balancer or DNS (hard-coding a set of IPs). The consumer wouldn't have any way to recover currently, as none of the configured IPs would be valid anymore.

Just making brokers accept a function isn't quite enough to solve that though. We would need to figure out when to call it. Currently we only use the seed brokers during startup, so we'd only call it once. We'd need to refresh the list of seed brokers maybe on crash or something.

In the mean time, I would really recommend either having a load balancer in front of your brokers (not to actually distribute load, since any connections afterwards will go directly to the broker) or to use DNS so that broker-1 always points to some broker, even if that broker might change over time.

[robodude666]

Regardless of using the seed broker to discover other kafka brokers, I'd still like this functionality for one reason:

• All currently known brokers may be offline.

Having this function would allow a centralized place to put all logic for fetching the list of brokers. It could be called during initialization, then after a connection is lost after a certain # of retries.

This could also address #593.