Skip to content

Commit

Permalink
update doc
Browse files Browse the repository at this point in the history
  • Loading branch information
zhenik committed Aug 9, 2019
1 parent b878079 commit 8a72629
Showing 1 changed file with 36 additions and 37 deletions.
73 changes: 36 additions & 37 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,62 +1,61 @@
[![Build Status](https://www.travis-ci.org/sysco-middleware/kafka-client-collector.svg?branch=master)](https://www.travis-ci.org/sysco-middleware/kafka-client-collector)
# Kafka clients JMX collector
Kafka clients JMX collector is [Custom collector](https://github.com/prometheus/client_java#custom-collectors), for [prometheus/client_java](https://github.com/prometheus/client_java#custom-collectors).
[![Maven metadata URI](https://img.shields.io/maven-metadata/v/http/central.maven.org/maven2/no/sysco/middleware/prometheus/kafka-client-collector/maven-metadata.xml.svg)](https://repo1.maven.org/maven2/no/sysco/middleware/prometheus/kafka-client-collector)

`Kafka clients JMX collector` does proxy JMX metrics from kafka clients.

## Versioning
todo:
0.6.0-2.1.1 -> {prometheus.kafka-clients}
# Kafka client collector
Kafka client collector is an implementation of [Prometheus custom collector](https://github.com/prometheus/client_java#custom-collectors),
for collecting JMX metrics from kafka clients.

## Usage
Import dependency.
```xml
<dependency>
<groupId>no.sysco.middleware.prometheus</groupId>
<artifactId>kafka-client-collector</artifactId>
<version>${version}</version>
<version>0.0.2</version>
</dependency>
```

Use `KafkaClientsJmxExports` to initialize collectors for kafka-clients JMX metrics to conveniently register them.
Use `KafkaClientsJmxExports` to initialize collectors for kafka client's JMX metrics to conveniently register them.
```java
KafkaClientsJmxExports.initialize(kafkaProducer);
HTTPServer server = new HTTPServer(8081);
Runtime.getRuntime().addShutdownHook(new Thread(server::stop));

KafkaClientsJmxExports.initialize(kafkaProducer1);
```
## Metric format
```
JMX example
kafka.producer:type=producer-metrics,client-id="dasf-gdfgd-dfgd-31",waiting-threads="5"
## Idea
Provide availability to expose:
- kafka clients jmx related metrics ([KafkaClientsJmxExports.initialize(kafkaClient);](./src/main/java/no/sysco/middleware/prometheus/kafka/KafkaClientsJmxExports.java))
- custom metrics (developer's responsibility)
- resource utilization metrics ([DefaultExports.initialize();](https://github.com/prometheus/client_java/blob/master/simpleclient_hotspot/src/main/java/io/prometheus/client/hotspot/DefaultExports.java))
Will be exposed as
producer_metrics_waiting_threads {cliend-id="dasf-gdfgd-dfgd-31"} 5.0
```
JMX domain (kafka.producer - example above) is not present in Prometheus format.

## Metrics types
Metrics with Attributes. [Reference](https://github.com/prometheus/jmx_exporter/pull/305/commits/92a6eb106e84cd441ba9b6123132395738d6acd6)
Name of `metrics group` (prometheus context) or `metric type` (jmx context)
## Metrics types (group)
### Producer:
* `app-info` @deprecated = common clients metrics
* `producer-metrics` = common clients metrics + only producer related metrics
* `producer-topic-metrics` = only producer related metrics
* `producer-node-metrics` = common clients metrics

* `app-info` @deprecated
* `producer-metrics`
* `producer-topic-metrics`
* `producer-node-metrics`
### Consumer:
* `app-info` @deprecated = common clients metrics
* `consumer-metrics` = common clients metrics
* `consumer-coordinator-metrics` - consumer group metrics
* `consumer-fetch-manager-metrics` = fetch-manager metrics + per topic + per partition
* `consumer-node-metrics` = common clients metrics

* `app-info` @deprecated
* `consumer-metrics`
* `consumer-coordinator-metrics`
* `consumer-fetch-manager-metrics`
* `consumer-node-metrics`
### Stream:
Stream contains metrics from domains `kafka.producer`, `kafka.consumer`, `kafka.admin.client` and own set of metrics
such as :
* `app-info` @deprecated = common clients metrics
* `stream-metrics` = only stream related metrics [INFO lvl]
* `stream-task-metrics` = stream task related metrics [DEBUG lvl]
* `stream-processor-node-metrics` = stream processor related metrics [DEBUG lvl]
* `stream-[store-scope]-metrics` = stream store related metrics [DEBUG lvl]
* `stream-record-cache-metrics` = stream record cache related metrics [DEBUG lvl]
* `stream-buffer-metrics` = stream buffer related metrics [DEBUG lvl]
* `app-info` @deprecated
* `stream-metrics` [INFO lvl]
* `stream-task-metrics` [DEBUG lvl]
* `stream-processor-node-metrics` [DEBUG lvl]
* `stream-[store-scope]-metrics` [DEBUG lvl]
* `stream-record-cache-metrics` [DEBUG lvl]
* `stream-buffer-metrics` [DEBUG lvl]

### References
- [More examples](https://github.com/sysco-middleware/kafka-client-collector-examples)
- [Issue 305: Add kafka client example config](https://github.com/prometheus/jmx_exporter/pull/305#issuecomment-412851484)
- [Issue 400: does the client automatically publish jmx mertices (heap size memory, thread number)?](https://github.com/prometheus/client_java/issues/400)
- [Blog post: JMX monitoring + Java custom metrics.](https://sysdig.com/blog/jmx-monitoring-custom-metrics/)
Expand Down

0 comments on commit 8a72629

Please sign in to comment.