Skip to content

Latest commit

 

History

History
93 lines (83 loc) · 4.32 KB

README.md

File metadata and controls

93 lines (83 loc) · 4.32 KB

Build Status Maven metadata URI

Kafka client collector

Kafka client collector is an implementation of Prometheus custom collector, for collecting JMX metrics from kafka clients.

Usage

<dependency>
    <groupId>no.sysco.middleware.prometheus</groupId>
    <artifactId>kafka-client-collector</artifactId>
    <version>0.0.2</version>
</dependency>

Prometheus http server

Use KafkaClientsJmxExports to initialize collectors for kafka client's JMX metrics to conveniently register them.

HTTPServer server = new HTTPServer(8081);
Runtime.getRuntime().addShutdownHook(new Thread(server::stop));

KafkaClientsJmxExports.initialize(kafkaProducer1);

Armeria http server

Add armeria dependency, than

/** init health-check, config, metrics */
final CollectorRegistry collectorRegistry = CollectorRegistry.defaultRegistry;
new ClientsJmxCollector(kafkaProducer).register(collectorRegistry);

Server server = new ServerBuilder().http(8085)
    .service("/", (ctx, req) -> HttpResponse.of("OK"))
    .service("/config", (ctx, req) -> HttpResponse.of(appConfig.properties.toString()))
    .service("/metrics", (ctx, req) -> {
      final ByteArrayOutputStream stream = new ByteArrayOutputStream();
      try (OutputStreamWriter writer = new OutputStreamWriter(stream)) {
        TextFormat.write004(writer, collectorRegistry.metricFamilySamples());
      }
      return HttpResponse.of(HttpStatus.OK, CONTENT_TYPE_004, stream.toByteArray());
    })
    .build();
CompletableFuture<Void> future = server.start();
// Wait until the server is ready.
future.join();

Follow full example

Metric format

JMX example
kafka.producer:type=producer-metrics,client-id="dasf-gdfgd-dfgd-31",waiting-threads="5"

Will be exposed as
producer_metrics_waiting_threads {cliend-id="dasf-gdfgd-dfgd-31"} 5.0

JMX domain (kafka.producer - example above) is not present in Prometheus format.

Metrics types (group)

Producer:

  • app-info @deprecated
  • producer-metrics
  • producer-topic-metrics
  • producer-node-metrics

Consumer:

  • app-info @deprecated
  • consumer-metrics
  • consumer-coordinator-metrics
  • consumer-fetch-manager-metrics
  • consumer-node-metrics

Stream:

Stream contains metrics from domains kafka.producer, kafka.consumer, kafka.admin.client and own set of metrics such as :

  • app-info @deprecated
  • stream-metrics [INFO lvl]
  • stream-task-metrics [DEBUG lvl]
  • stream-processor-node-metrics [DEBUG lvl]
  • stream-[store-scope]-metrics [DEBUG lvl]
  • stream-record-cache-metrics [DEBUG lvl]
  • stream-buffer-metrics [DEBUG lvl]

References