OpenTelemetry Instrumentation for Java

Join the discussions!

• Watch this repo 👁️
• Join the Gitter channel
• Join the weekly meeting

Introduction

This project provides a Java agent JAR that can be attached to any Java 7+ application and dynamically injects bytecode to capture telemetry from a number of popular libraries and frameworks. The telemetry data can be exported in a variety of formats. In addition, the agent and exporter can be configured via command line arguments or environment variables. The net result is the ability to gather telemetry data from a Java application without code changes.

Getting Started

Download the latest version.

This package includes the instrumentation agent, instrumentations for all supported libraries and all available data exporters. This provides completely automatic out of the box experience.

The instrumentation agent is enabled using the -javaagent flag to the JVM.

java -javaagent:path/to/opentelemetry-javaagent-all.jar \
     -jar myapp.jar

By default OpenTelemetry Java agent uses OTLP exporter configured to send data to OpenTelemetry collector at localhost:55680.

Configuration parameters are passed as Java system properties (-D flags) or as environment variables (see below for full list). For example:

java -javaagent:path/to/opentelemetry-javaagent-all.jar \
     -Dota.exporter=zipkin
     -jar myapp.jar

External exporter jar can be specified via ota.exporter.jar system property:

java -javaagent:path/to/opentelemetry-javaagent-all.jar \
     -Dota.exporter.jar=path/to/external-exporter.jar
     -jar myapp.jar

Configuration parameters (subject to change!)

Note: These parameter names are very likely to change over time, so please check back here when trying out a new version! Please report any bugs or unexpected behavior you may find.

Jaeger exporter

A simple wrapper for the Jaeger exporter of opentelemetry-java. It currently only supports gRPC as its communications protocol.

System property	Environment variable	Purpose
ota.exporter=jaeger	OTA_EXPORTER=jaeger	To select Jaeger exporter
JAEGER_ENDPOINT	JAEGER_ENDPOINT	The Jaeger endpoint to connect to. Currently only gRPC is supported.
JAEGER_SERVICE_NAME	JAEGER_SERVICE_NAME	The service name of this JVM instance

Zipkin exporter

A simple wrapper for the Zipkin exporter of opentelemetry-java. It POSTs json in Zipkin format to a specified HTTP URL.

System property	Environment variable	Purpose
ota.exporter=zipkin	OTA_EXPORTER=zipkin	To select Zipkin exporter
otel.zipkin.endpoint	OTEL_ZIPKIN_ENDPOINT	The Zipkin endpoint to connect to. Currently only HTTP is supported.
otel.zipkin.service.name	OTEL_ZIPKIN_SERVICE_NAME	The service name of this JVM instance

OTLP exporter

A simple wrapper for the OTLP exporter of opentelemetry-java.

System property	Environment variable	Purpose
ota.exporter=otlp (default)	OTA_EXPORTER=otlp	To select OpenTelemetry exporter (default)
otel.otlp.endpoint	OTEL_OTLP_ENDPOINT	The OTLP endpoint to connect to.
otel.otlp.use.tls	OTEL_OTLP_USE_TLS	To use or not TLS, default is false.
otel.otlp.metadata	OTEL_OTLP_METADATA	The key-value pairs separated by semicolon to pass as request metadata.
otel.otlp.span.timeout	OTEL_OTLP_SPAN_TIMEOUT	The max waiting time allowed to send each span batch, default is 1000.

Logging exporter

The logging exporter simply prints the name of the span along with its attributes to stdout. It is used mainly for testing and debugging.

System property	Environment variable	Purpose
ota.exporter=logging	OTA_EXPORTER=logging	To select logging exporter
ota.exporter.logging.prefix	OTA_EXPORTER_LOGGING_PREFIX	An optional string that is printed in front of the span name and attributes.

Batch span processor

System property	Environment variable	Purpose
otel.bsp.schedule.delay	OTEL_BSP_SCHEDULE_DELAY	The interval in milliseconds between two consecutive exports (default: 5000)
otel.bsp.max.queue	OTEL_BSP_MAX_QUEUE	Maximum queue size (default: 2048)
otel.bsp.max.export.batch	OTEL_BSP_MAX_EXPORT_BATCH	Maximum batch size (default: 512)
otel.bsp.export.timeout	OTEL_BSP_EXPORT_TIMEOUT	Maximum allowed time in milliseconds to export data (default: 30000)
otel.bsp.export.sampled	OTEL_BSP_EXPORT_SAMPLED	Whether only sampled spans should be exported (default: true)

Trace config

System property	Environment variable	Purpose
otel.config.sampler.probability	OTEL_CONFIG_SAMPLER_PROBABILITY	Sampling probability between 0 and 1 (default: 1)
otel.config.max.attrs	OTEL_CONFIG_MAX_ATTRS	Maximum number of attributes per span (default: 32)
otel.config.max.events	OTEL_CONFIG_MAX_EVENTS	Maximum number of events per span (default: 128)
otel.config.max.links	OTEL_CONFIG_MAX_LINKS	Maximum number of links per span (default: 32)
otel.config.max.event.attrs	OTEL_CONFIG_MAX_EVENT_ATTRS	Maximum number of attributes per event (default: 32)
otel.config.max.link.attrs	OTEL_CONFIG_MAX_LINK_ATTRS	Maximum number of attributes per link (default: 32)

Interval metric reader

System property	Environment variable	Purpose
otel.imr.export.interval	OTEL_IMR_EXPORT_INTERVAL	The interval in milliseconds between pushes to the exporter (default: 60000)

Customizing the OpenTelemetry SDK

This is highly advanced behavior and still in the prototyping phase. It may change drastically or be removed completely. Use with caution

The OpenTelemetry API exposes SPI hooks for customizing its behavior, such as the Resource attached to spans or the Sampler.

Because the auto instrumentation runs in a separate classpath than the instrumented application, it is not possible for customization in the application to take advantage of this customization. In order to provide such customization, you can provide the path to a JAR file including an SPI implementation using the system property ota.initializer.jar. Note that this JAR will need to shade the OpenTelemetry API in the same way as the agent does. The simplest way to do this is to use the same shading configuration as the agent from here. In addition, you will have to specify the io.opentelemetry.auto.shaded.io.opentelemetry.trace.spi.TraceProvider to the name of the class that implements the SPI.

Supported Java libraries and frameworks

Library/Framework	Versions
Akka HTTP	10.0+
Apache HttpAsyncClient	4.0+
Apache HttpClient	2.0+
AWS SDK	1.11.x and 2.2.0+
Cassandra Driver	3.0+ (not including 4.x yet)
Couchbase Client	2.0+ (not including 3.x yet)
Dropwizard Views	0.7+
Elasticsearch API	2.0+ (not including 7.x yet)
Elasticsearch REST Client	5.0+
Finatra	2.9+
Geode Client	1.4+
Google HTTP Client	1.19+
Grizzly	2.0+ (disabled by default, see below)
Grizzly Client	1.9+
gRPC	1.5+
Hibernate	3.3+
HttpURLConnection	Java 7+
Hystrix	1.4+
java.util.logging	Java 7+
JAX-RS	0.5+
JAX-RS Client	2.0+
JDBC	Java 7+
Jedis	1.4+
Jetty	8.0+
JMS	1.1+
JSP	2.3+
Kafka	0.11+
khttp	0.1.0+
Kubernetes	7.0.0+
Lettuce	4.0+
Log4j	1.1+
Logback	1.0+
MongoDB Drivers	3.3+
Netty	3.8+
OkHttp	3.0+
Play	2.3+ (not including 2.8.x yet)
Play WS	1.0+
RabbitMQ Client	2.7+
Ratpack	1.4+
Reactor	3.1+
Rediscala	1.8+
RMI	Java 7+
RxJava	1.0+
Servlet	2.2+
Spark Web Framework	2.3+
Spring Data	1.8+
Spring Scheduling	3.1+
Spring Web MVC	3.1+
Spring Webflux	5.0+
Spymemcached	2.12+
Twilio	6.6+
Vert.x	3.0+
Vert.x RxJava2	3.5+

Disabled instrumentations

Some instrumentations can produce too many spans and make traces very noisy. For this reason the following instrumentations are disabled by default:

• jdbc-datasource which creates spans whenever java.sql.DataSource#getConnection method is called.
• servlet-filter which creates spans around Servlet Filter methods.
• servlet-service which creates spans around Servlet methods.

To enable them, add ota.integration.<name>.enabled system property: -Dota.integration.jdbc-datasource.enabled=true

Grizzly instrumentation

Whenever you use Grizzly for Servlet-based applications, you get better experience from Servlet-specific support. As these two instrumentations conflict with each other, more generic instrumentation for Grizzly http server is disabled by default. If needed, you can enable it by add the following system property: -Dota.integration.grizzly.enabled=true

Manually instrumenting

⚠️ starting with 0.6.0, and prior to version 1.0.0, opentelemetry-javaagent-all.jar only supports manual instrumentation using the opentelemetry-api version with the same version number as the Java agent you are using. Starting with 1.0.0, the Java agent will start supporting multiple (1.0.0+) versions of opentelemetry-api.

You can use the OpenTelemetry getTracer or the @WithSpan annotation to manually instrument your Java application.

Configure the OpenTelemetry getTracer

OpenTelemetry offers a tracer to easily enable custom instrumentation throughout your application. See the OpenTelemetry Java QuickStart for an example of how to configure it.

Configure a WithSpan annotation

If you want to configure custom instrumentation and don't want to use the OpenTelemetry getTracer and API directly, configure a @WithSpan annotation. Add the trace annotation to your application's code:

import io.opentelemetry.contrib.auto.annotations.WithSpan;

public class MyClass {
  @WithSpan
  public void MyLogic() {
      <...>
  }
}

Each time the application invokes the annotated method, it creates a span that denote its duration and provides any thrown exceptions.

Configuration

The @WithSpan annotation requires code changes to implement. You can disable the annotation at runtime via the exclude configuration or environment variables:

System property	Environment variable	Purpose
trace.classes.exclude	TRACE_CLASSES_EXCLUDE	Exclude classes with the @WithSpan annotation
trace.methods.exclude	TRACE_METHODS_EXCLUDE	Exclude methods with the @WithSpan annotation

Troubleshooting

To turn on the agent's internal debug logging:

-Dio.opentelemetry.auto.slf4j.simpleLogger.defaultLogLevel=debug

Note these logs are extremely verbose. Enable debug logging only when needed. Debug logging negatively impacts the performance of your application.

Roadmap to 1.0 (GA)

It is our goal to release 1.0 (GA) of the auto-instrumentation agent during the first wave of OpenTelemetry 1.0 (GA) releases, along with as many manual instrumentation libraries as possible.

High-level roadmap:

• Conform with all OpenTelemetry specifications
  • Implement all applicable semantic attributes
    • Clearly document additional attributes not defined in specification
  • Support standard configuration properties (e.g. exporters, propagators, samplers)
  • Capture standard metrics (still TBD, e.g. opentelemetry-specification#522)
  • See issues with label specification
• Great documentation
  • See issues with label documentation
• Build out manual instrumentation libraries for existing auto-instrumentation
  • Share code and tests between manual and auto-instrumentation
  • See issue #45
• Good support for vendors to extend the agent
  • Including ability for any user to write their own auto-instrumentation
  • See issues with label packaging
• Better smoke test harness and more smoke tests
  • See issue #298
• Benchmarking and tuning (both runtime and startup)
• Address sporadic test failures
  • See issues with label sporadic test failure
• Speed up CI build feedback