Kafka client collector

Kafka client collector is an implementation of Prometheus custom collector, for collecting JMX metrics from kafka clients.

Usage

<dependency>
    <groupId>no.sysco.middleware.prometheus</groupId>
    <artifactId>kafka-client-collector</artifactId>
    <version>0.0.2</version>
</dependency>

Use KafkaClientsJmxExports to initialize collectors for kafka client's JMX metrics to conveniently register them.

HTTPServer server = new HTTPServer(8081);
Runtime.getRuntime().addShutdownHook(new Thread(server::stop));

KafkaClientsJmxExports.initialize(kafkaProducer1);

Metric format

JMX example
kafka.producer:type=producer-metrics,client-id="dasf-gdfgd-dfgd-31",waiting-threads="5"

Will be exposed as
producer_metrics_waiting_threads {cliend-id="dasf-gdfgd-dfgd-31"} 5.0

JMX domain (kafka.producer - example above) is not present in Prometheus format.

Metrics types (group)

Producer:

• app-info @deprecated
• producer-metrics
• producer-topic-metrics
• producer-node-metrics

Consumer:

• app-info @deprecated
• consumer-metrics
• consumer-coordinator-metrics
• consumer-fetch-manager-metrics
• consumer-node-metrics

Stream:

Stream contains metrics from domains kafka.producer, kafka.consumer, kafka.admin.client and own set of metrics such as :

• app-info @deprecated
• stream-metrics [INFO lvl]
• stream-task-metrics [DEBUG lvl]
• stream-processor-node-metrics [DEBUG lvl]
• stream-[store-scope]-metrics [DEBUG lvl]
• stream-record-cache-metrics [DEBUG lvl]
• stream-buffer-metrics [DEBUG lvl]

References

• More examples
• Issue 305: Add kafka client example config
• Issue 400: does the client automatically publish jmx mertices (heap size memory, thread number)?
• Blog post: JMX monitoring + Java custom metrics.
• Kafka metrics reporter
• IMO - Good example how to make custom collector
• Approaches (POCs) with org.apache.kafka.common.metrics.MetricReporter
  • kafka-prometheus-metric-reporter
  • kafka-client-monitoring-poc
• Micrometer [PR WIP]