[improve][pip] PIP-438: add namespace topics limit metric

[alexandrebrg]

Motivation

We recently came with an issue where namespaces were reaching their limits, errors are lifted to producers and cannot be fixed by them (need either to delete topics or increase namespace limit). This PR creates a new metric to give topics limit for each namespace loaded in a broker. It will allow to correlate the current topic count (already existing metric) and the limit to determine the saturation of a namespace.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository

[lhotari]

This might require some more changes since topic stats contains only the topics that are currently loaded in the broker.

Since this is a cluster wide limit, it would be unnecessary work for each broker in the cluster to calculate the current amount of topics that are actually created in the namespace. One possible solution would be to handle this on the "leader broker" so that it's the only broker that creates these statistics.

The current solution for finding out the number of topics in the namespace might not be optimal. One possible solution would be to extend the MetadataStore interface to contain a method "getNumberOfChildren" which would use an optimal method for the particular MetadataStore implementation to find the number of child nodes. For example, on Zookeeper, there's a field on the parent ZNode that tells how many child znodes there are. This would be an efficient way at least on Zookeeper to find out how many topics there are in a namespace.

update: Since there's MetadataCache layer to list the topics in the namespace, this might not be a significant bottleneck. However, it would lead to the situation where all namespaces would get listed on the leader broker and kept in memory. That could be a problem in some cases and therefore having a way to get the number of children directly would be better. For MetadataStore implementation that don't support getting the number of child nodes, the cache could be used.

[alexandrebrg]

I believe this PIP might have been misunderstood, we are looking to implement a metric for each namespace where it would give insight on the maximum number of topics that can be created in a namespace (which is either a namespace policy related value or broker default configuration value).

I understand that a namespace can be bundled and the namespace can be scattered between multiple brokers, isn't there a way to determine a "leader" broker for a namespace based on the topics loaded ? Or any metadata related to it ? (e.g. bundleId if it exists?)