Expose cluster metadata in client(s)

[jlandersen]

Hi,

I maintain a Kafka extension for Visual Studio Code (https://marketplace.visualstudio.com/items?itemName=jeppeandersen.vscode-kafka), currently based on kafka-node. I am looking into migrating to KafkaJS, and found a few things missing before I can proceed.

The most urgent thing for me, is accessing cluster metadata (aside from topics). I might be wrong, but it does not seem to be possible, at this time, to access brokers (and potentially connection status) in the cluster (basically what is kept in this module https://github.com/tulios/kafkajs/blob/master/src/cluster/index.js). Is there a reason for not exposing the cluster data? Or did I completely miss some way to access it?

[Nevon]

Cool extension!

So far we have considered the cluster API to be internal. Our use-cases have been consuming and producing messages, plus some admin APIs, so anything to do with cluster operations has been considered an implementation detail that our users don't need to worry about.

What kind of cluster data and what kind of operations do you need to be able to make on the brokers? I haven't tried your extension out yet, but from the screenshots I only see that it lists brokers, but I can't really see what you can do with the brokers.

I'm not inclined to expose the cluster itself to the user, as that would make it rife for misuse. We also replace the cluster instance in some cases, so it would be really easy to get stale references. Furthermore, just looking at the metadata in the cluster is not sufficient for you use-case, as the metadata is not guaranteed to contain all of the brokers - only the brokers that we have needed to connect to so far.

I would be more inclined to expose a describeCluster method on the admin client, same as the Java client does, which is basically just a metadata request for all topics. I think this would be sufficient for what you need.

[jlandersen]

I wouldn't expect you to expose the full Cluster module just to be clear. Listing brokers (and connection status) is enough for me :-) The extension doesen't do any operations on brokers and I am not planning to add any. So a describeCluster method is definitely good for me.

I dont't mind adding it myself within the next couple of weeks. I was planning to submit a fix to describe configs in the near future as well (broker configs requests need to be sent to the specific broker, not controller).

[Nevon]

I took an initial stab at it. Currently it pretty much mirrors the Java API, although I went with the term broker over node, since we don't really expose the concept of a Node to the user currently (beyond the nodeId).

One addition over the Java API is that we now also expose a connected property on each broker.

[jlandersen]

Looks good - thanks! Once this change is out I will start some migration work. I hope to contribute where I can to fill in any other blanks :)

[Nevon]

Once the PR is reviewed and merged, there will be a pre-release version out that you can use to start your integration work. Feel free to open more issues and/or drop into our Slack channel if you run into any issues.

[jlandersen]

Thanks for the fast turnaround!

[jlandersen]

@Nevon would you happen to have any indication of when this might be pushed (1.12.0 seems to still be the one out)