Add kafka client example config

[saada]

The current example are mostly for kafka brokers. Monitoring kafka clients is instrumental. The hour spent putting this config together can save others a lot of time.

[brian-brazil]

Do you have sample output?

[brian-brazil]

Counters should end in _total

[brian-brazil]

The other question would be, is there a way to get this data at the Java level that isn't JMX? This may be better done as a collector.

[agolomoodysaada]

You raise a great point @brian-brazil . Any pointers on writing collectors of this nature? or pulling out mbeans out of kafka client?

I found this https://github.com/prometheus/client_java#custom-collectors but I'm not sure how to extract metrics directly from Kafka. Perhaps this is relevant?

I'll attempt using KafkaMetricsReporter and see what I come up with.

[brian-brazil]

Yes, https://github.com/apache/kafka/blob/2.0.0/clients/src/main/java/org/apache/kafka/common/metrics/MetricsReporter.java is what to look at. By writing a custom collector you can use this with any direct instrumentation you also have inside your application.

[mliang2]

This doens't parse the metrics name properly. Example output w/ client v 1.1.1:

kafka_consumer_consumer{client_id="client-1",} 6.11055928E9
kafka_consumer_consumer{client_id="client-1",} 7.734885164556962E7
kafka_consumer_consumer{client_id="client-1",} 8075.0

kafka_consumer_consumer_coordinator{client_id="client-1",} 0.0
kafka_consumer_consumer_coordinator{client_id="client-1",} 0.029547334830398297
kafka_consumer_consumer_coordinator{client_id="client-1",} 0.029672710008605087

kafka_consumer_consumer_fetch_manager{client_id="client-1",} 63.0
kafka_consumer_consumer_fetch_manager{client_id="client-1",} 8149.0
kafka_consumer_consumer_fetch_manager{client_id="client-1, topic=topic-1",} 0.05937184587068812

kafka_consumer_consumer_node{client_id="client-1, node-id=node-2147483516",} 28.43460041903622
kafka_consumer_consumer_node{client_id="client-1, node-id=node-2147483516",} 767.0
kafka_consumer_consumer_node{client_id="client-1, node-id=node-2147483516",} 767.0

kafka_producer_producer{client_id="producer-1",} 7.5
kafka_producer_producer{client_id="producer-1",} 92.5
kafka_producer_producer{client_id="producer-1",} 99.0

kafka_producer_producer_node{client_id="producer-1, node-id=node--1",} 588.0
kafka_producer_producer_node{client_id="producer-1, node-id=node--1",} 588.0
kafka_producer_producer_node{client_id="producer-1, node-id=node--1",} 66.0

kafka_producer_producer_topic{client_id="producer-1, topic=topic-1",} 0.0
kafka_producer_producer_topic{client_id="producer-1, topic=topic-1",} 0.0
kafka_producer_producer_topic{client_id="producer-1, topic=topic-1",} 0.05443361820260193

[LiamClarkeNZ]

I'm looking into Prometheus for monitoring Kafka producers across the system to identify any misconfiguration at a higher level, and JMX is the most simplest way (from an ops POV) to gather producer metrics across an entire JVM. So I think an examplke of that would be ideal to have on hand.

@agolomoodysaada If you're still looking into this, the consumers and producers take a configuration item "metric.reporters" that is a list of class names that implement MetricsReporter, that's how you extract the metrics from the producer/consumer, but note that it's per instance.

[zhenik]

Hi guys,
I have recently released custom collector for kafka clients. You can take a look here

Also you can find here examples how to access all metrics from Kafka-clients, scrape it via implements Runnable interface - http://cloudurable.com/blog/kafka-tutorial-kafka-producer-advanced-java-examples/index.html

[brian-brazil]

Looks like a better way has been found.