Does zilla supports datadog APM onboarding ?

[vamsi260]

Does zilla supports datadog APM onboarding ? Does the base image has full JDK ? There's an error with missing java module while i try to download dd java agent & include in the image itself. let me know if there is a way to include otherthan this approach. Thanks in advance.

Could not find agent library instrument on the library path, with error: libinstrument.so: cannot open shared object file: No such file or directory

Module java.instrument may be missing from runtime image.

Error occurred during initialization of VM

Here is my Dockerfile.

FROM ghcr.io/aklivity/zilla:0.9.98
ENV ZILLA_VERSION=0.9.98

RUN apt-get update -qq && \
    apt-get install -y \
    curl \
    wget \
    net-tools \
    netcat \
    ca-certificates \
    zip \
    openjdk-17-jdk \
    python3 \
    python3-pip && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* && \
    pip3 install --no-cache-dir awscli 

RUN wget -O /opt/zilla/dd-java-agent.jar https://dtdg.co/latest-java-tracer

# Create necessary directories and download files
RUN mkdir -p /etc/zilla 

VOLUME ["/etc/zilla"]

# Copy the Zilla configuration file into the image
COPY files/zilla.yaml /etc/zilla/zilla.yaml
COPY files/zilla /opt/zilla/zilla

# Set the entrypoint for the application
ENTRYPOINT ["/opt/zilla/zilla"]

# Default command to start Zilla
CMD ["start", "-c", "/etc/zilla/zilla.yaml", "-v"]

Passed the below variables from zilla executable script

DD_JAVA_AGENT_JAR="/opt/zilla/dd-java-agent.jar"
JAVA_OPTIONS="$JAVA_OPTIONS -javaagent:$DD_JAVA_AGENT_JAR"

[vordimous]

Does the base image has full JDK ?

The base image for the runtime container is ubuntu:jammy-20240530.

@vamsi260 I have rebuilt the Zilla base image using eclipse-temurin:17-jdk-jammy and still get the java.lang.module.FindException: Module java.instrument not found error. This means Zilla doesn't support the Datadog java-tracer out of the box.

[vamsi260]

I presume, enabling the Java tracer would leverage the DD APM onboarding. That actually can help to assess the 360 degree performance of the application.

[jfallows]

@vamsi260 Thanks for the detailed instructions, we were able to reproduce the error you encountered on startup when attempting to instrument.

With the changes from PR #1321 in place via a local build of develop branch, zilla now outputs the following.

wget -O /tmp/dd-java-agent.jar https://dtdg.co/latest-java-tracer

docker run -p 7114:7114 \
  -e "JAVA_OPTIONS=-javaagent:/opt/zilla/dd-java-agent.jar" \
  -v /tmp/dd-java-agent.jar:/opt/zilla/dd-java-agent.jar \
  -v "$(pwd)"/files/zilla.yaml:/etc/zilla/zilla.yaml \
  ghcr.io/aklivity/zilla:develop-SNAPSHOT \
  start -v -e

[dd.trace 2024-11-05 23:43:12:028 +0000] [dd-telemetry] WARN datadog.telemetry.TelemetryRouter - Got FAILURE sending telemetry request to http://localhost:8126/telemetry/proxy/api/v2/apmtelemetry.
[dd.trace 2024-11-05 23:43:12:488 +0000] [dd-task-scheduler] INFO datadog.trace.agent.core.StatusLogger - DATADOG TRACER CONFIGURATION {"version":"1.42.0~6eac0b8465","os_name":"Linux","os_version":"6.10.11-linuxkit","architecture":"aarch64","lang":"jvm","lang_version":"22.0.2","jvm_vendor":"Eclipse Adoptium","jvm_version":"22.0.2+9","java_class_version":"66.0","http_nonProxyHosts":"null","http_proxyHost":"null","enabled":true,"service":"io.aklivity.zilla.runtime.command/io.aklivity.zilla.runtime.command.internal.ZillaMain","agent_url":"http://localhost:8126","agent_error":true,"debug":false,"trace_propagation_style_extract":["datadog","tracecontext"],"trace_propagation_style_inject":["datadog","tracecontext"],"analytics_enabled":false,"priority_sampling_enabled":true,"logs_correlation_enabled":true,"profiling_enabled":false,"remote_config_enabled":true,"debugger_enabled":false,"debugger_exception_enabled":false,"appsec_enabled":"ENABLED_INACTIVE","rasp_enabled":true,"telemetry_enabled":true,"telemetry_dependency_collection_enabled":true,"telemetry_log_collection_enabled":true,"dd_version":"","health_checks_enabled":true,"configuration_file":"no config file present","runtime_id":"a907212a-9eb6-4d3f-b9ef-6b83ab13f584","logging_settings":{"levelInBrackets":false,"dateTimeFormat":"'[dd.trace 'yyyy-MM-dd HH:mm:ss:SSS Z']'","logFile":"System.err","configurationFile":"simplelogger.properties","showShortLogName":false,"showDateTime":true,"showLogName":true,"showThreadName":true,"defaultLogLevel":"INFO","warnLevelString":"WARN","embedException":false},"cws_enabled":false,"cws_tls_refresh":5000,"datadog_profiler_enabled":false,"datadog_profiler_safe":true,"datadog_profiler_enabled_overridden":false,"data_streams_enabled":false}
---
name: example
bindings:
  north_tcp_server:
    type: tcp
    kind: server
    options:
      host: 0.0.0.0
      port:
        - 7114
    exit: north_http_server
  north_http_server:
    type: http
    kind: server
    exit: north_echo_server
  north_echo_server:
    type: echo
    kind: server
telemetry:
  exporters:
    stdout_logs_exporter:
      type: stdout

started
^Cstopped

Please confirm that this is what you need.

We'll include this PR in the next zilla release after it is merged.

[vamsi260]

Thanks @jfallows . But the error is different in my case. missing java.instrument module(complete exception below).
Could not find agent library instrument on the library path, with error: libinstrument.so: cannot open shared object file: No such file or directory Module java.instrument may be missing from runtime image. Error occurred during initialization of VM

If we include this dd java tracer in zilla base image, we can leverage the Datadog APM onboarding & bypass the handling exception of this instrument module.

[jfallows]

@vamsi260 right, the PR now includes the java.instrument module via the --instrument argument to zpmw in the zilla Dockerfile.

The example shows the instrumentation agent initializing properly without error using the local develop-SNAPSHOT building including that change.

The module not found error you reported reproduces with the current zilla release but not with the local develop-SNAPSHOT build.

The next zilla release will include the PR changes.

In the meantime, before the next official zilla release, you are able to verify that the behavior is as you expect with a custom Dockerfile using the zilla develop-SNAPSHOT tag from a local zilla build.

Please let us know if you have any questions.

[vamsi260]

Sure @jfallows . let me give it a try with develop-SNAPSHOT version and get back to you with my findings.