Traces do not appear in Datadog Web UI when AOT Cache is enabled,

Tracer Version(s)

1.58.2(, 1.57.0, 1.58.0 too)

Java Version(s)

25

JVM Vendor

Eclipse Adoptium / Temurin

Bug Report

Behavior

When the application is started with AOT Cache enabled, traces do not appear in Datadog Web UI.
Using the same container image and the same JAR, traces do appear if AOT Cache is disabled.
Also, if AOT Cache is enabled but AOT Class Linking is disabled, traces do appear.

With DD_TRACE_DEBUG=true, the following log appears much more often in the NG case than in the OK cases, but it is unclear if this is the root cause:
DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=xxx.xxx.Xxx instrumentation.target.context=yyy.yyy.Yyy->zzz.zzz.Zzz

From the logs, sending to the Datadog Agent seems successful:

[dd-trace-processor] EXCLUDE_TELEMETRY datadog.trace.agent.common.writer.ddagent.DDAgentApi - Success while sending 8 (size=6KB) traces. Total: 18, Received: 18, Sent: 18, Failed: 0.
[dd-trace-processor] DEBUG datadog.trace.agent.common.sampling.RateByServiceTraceSampler - Update service sampler rates: http://dd-agent:8126/v0.4/traces -> {rate_by_service={service:,env:=1.0}}
[dd-trace-processor] DEBUG datadog.trace.agent.common.writer.PayloadDispatcherImpl - Successfully sent 8 traces to the API

I would appreciate any guidance on potential causes or possible workarounds.

Expected Behavior

Traces appear in Datadog Web UI

Reproduction Code

Case 1 (NG)

The application starts, but traces do not appear.

java -javaagent:/dd-java-agent.jar \
  -Xlog:aot \
  -XX:AOTCache=/workspace/application.aot \
  -jar application.jar

`Failed to add context-store field` output

[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.ProxyStatement instrumentation.target.context=java.sql.Statement->datadog.trace.bootstrap.instrumentation.jdbc.DBQueryInfo
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.ProxyLeakTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=software.amazon.jdbc.util.monitoring.AbstractMonitor instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.HikariPool$KeepaliveTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.HikariPool$MaxLifetimeTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.HikariPool$HouseKeeper instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.apache.tomcat.util.net.SocketWrapperBase$OperationState instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.apache.tomcat.util.net.Acceptor instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessorMonitor instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=io.sentry.backpressure.BackpressureMonitor instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=io.sentry.PreviousSessionFinalizer instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=io.sentry.MovePreviousSession instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=tools.jackson.databind.util.internal.PrivateMaxEntriesMap$AddTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=ch.qos.logback.classic.spi.LoggingEvent instrumentation.target.context=ch.qos.logback.classic.spi.ILoggingEvent->datadog.trace.bootstrap.instrumentation.api.AgentSpanContext
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.mysql.cj.jdbc.StatementImpl instrumentation.target.context=java.sql.Statement->datadog.trace.bootstrap.instrumentation.jdbc.DBQueryInfo
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.springframework.beans.factory.support.DisposableBeanAdapter instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.mysql.cj.jdbc.ConnectionImpl instrumentation.target.context=java.sql.Connection->datadog.trace.bootstrap.instrumentation.jdbc.DBInfo
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.mysql.cj.jdbc.AbandonedConnectionCleanupThread instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.ProxyConnection instrumentation.target.context=java.sql.Connection->datadog.trace.bootstrap.instrumentation.jdbc.DBInfo
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=jdk.internal.net.http.common.SequentialScheduler$SchedulableTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=jdk.internal.net.http.Http1Exchange$Http1Publisher$WriteTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=sun.rmi.transport.tcp.TCPTransport$AcceptLoop instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.util.concurrent.CompletableFuture$UniCompletion instrumentation.target.context=java.util.concurrent.CompletableFuture$UniCompletion->datadog.trace.bootstrap.instrumentation.java.concurrent.ConcurrentState
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.util.concurrent.FutureTask instrumentation.target.context=java.util.concurrent.RunnableFuture->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.util.concurrent.ThreadPoolExecutor$Worker instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.util.concurrent.ThreadPoolExecutor instrumentation.target.context=java.util.concurrent.ThreadPoolExecutor->java.lang.Boolean
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.net.HttpURLConnection instrumentation.target.context=java.net.HttpURLConnection->datadog.trace.bootstrap.instrumentation.httpurlconnection.HttpUrlState
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.util.concurrent.ForkJoinTask instrumentation.target.context=java.util.concurrent.ForkJoinTask->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.lang.VirtualThread instrumentation.target.context=java.lang.VirtualThread->datadog.trace.bootstrap.instrumentation.api.AgentScope
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.ProxyStatement instrumentation.target.context=java.sql.Statement->datadog.trace.bootstrap.instrumentation.jdbc.DBQueryInfo
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.ProxyLeakTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=software.amazon.jdbc.util.monitoring.AbstractMonitor instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.HikariPool$KeepaliveTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.HikariPool$MaxLifetimeTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.HikariPool$HouseKeeper instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.apache.tomcat.util.net.SocketWrapperBase$OperationState instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.apache.tomcat.util.net.Acceptor instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessorMonitor instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=io.sentry.backpressure.BackpressureMonitor instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=io.sentry.PreviousSessionFinalizer instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=io.sentry.MovePreviousSession instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=tools.jackson.databind.util.internal.PrivateMaxEntriesMap$AddTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=ch.qos.logback.classic.spi.LoggingEvent instrumentation.target.context=ch.qos.logback.classic.spi.ILoggingEvent->datadog.trace.bootstrap.instrumentation.api.AgentSpanContext
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.mysql.cj.jdbc.StatementImpl instrumentation.target.context=java.sql.Statement->datadog.trace.bootstrap.instrumentation.jdbc.DBQueryInfo
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.springframework.beans.factory.support.DisposableBeanAdapter instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.mysql.cj.jdbc.ConnectionImpl instrumentation.target.context=java.sql.Connection->datadog.trace.bootstrap.instrumentation.jdbc.DBInfo
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.mysql.cj.jdbc.AbandonedConnectionCleanupThread instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=jdk.internal.net.http.common.SequentialScheduler$SchedulableTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=jdk.internal.net.http.Http1Exchange$Http1Publisher$WriteTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=sun.rmi.transport.tcp.TCPTransport$AcceptLoop instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.util.concurrent.CompletableFuture$UniCompletion instrumentation.target.context=java.util.concurrent.CompletableFuture$UniCompletion->datadog.trace.bootstrap.instrumentation.java.concurrent.ConcurrentState
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.util.concurrent.FutureTask instrumentation.target.context=java.util.concurrent.RunnableFuture->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.util.concurrent.ThreadPoolExecutor$Worker instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.util.concurrent.ThreadPoolExecutor instrumentation.target.context=java.util.concurrent.ThreadPoolExecutor->java.lang.Boolean
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.net.HttpURLConnection instrumentation.target.context=java.net.HttpURLConnection->datadog.trace.bootstrap.instrumentation.httpurlconnection.HttpUrlState
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.util.concurrent.ForkJoinTask instrumentation.target.context=java.util.concurrent.ForkJoinTask->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.lang.VirtualThread instrumentation.target.context=java.lang.VirtualThread->datadog.trace.bootstrap.instrumentation.api.AgentScope

Case 2 (OK)

Traces appear when AOT Cache is not specified.

java -javaagent:/dd-java-agent.jar \
  -Xlog:aot \
  -jar application.jar

`Failed to add context-store field` output

[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.lang.VirtualThread instrumentation.target.context=java.lang.VirtualThread->datadog.trace.bootstrap.instrumentation.api.AgentScope
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.lang.VirtualThread instrumentation.target.context=java.lang.VirtualThread->datadog.trace.bootstrap.instrumentation.api.AgentScope

Case 3 (OK)

Traces appear when AOT Class Linking is disabled.

java -javaagent:/dd-java-agent.jar \
  -Xlog:aot \
  -XX:-AOTClassLinking \
  -XX:AOTCache=/workspace/application.aot \
  -jar application.jar

`Failed to add context-store field` output

[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.lang.VirtualThread instrumentation.target.context=java.lang.VirtualThread->datadog.trace.bootstrap.instrumentation.api.AgentScope
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.lang.VirtualThread instrumentation.target.context=java.lang.VirtualThread->datadog.trace.bootstrap.instrumentation.api.AgentScope

Added

Environment

Datadog
- dd-trace-java 1.58.2 / Agent 7.73.3
Kubernetes (Amazon EKS)

Eclipse Temurin 25

java --version
openjdk 25 2025-09-16 LTS
OpenJDK Runtime Environment Temurin-25+36 (build 25+36-LTS)
OpenJDK 64-Bit Server VM Temurin-25+36 (build 25+36-LTS, mixed mode, sharing)

Spring Boot 4.0.0
- hikariCP
- MySQL (Amazon Aurora)

@footaku

Hi @footaku - the -XX:AOT option is not recognized by the Eclipse/Temurin JVM, I assume you're using IBM/Semeru? Could you post the full version of the JVM to be sure?

I've confirmed that traces are received when using Eclipse/Temurin 25 with -XX:AOTCache so this is probably something specific to IBM Semeru's AOT, which is different to OpenJDK's.

To help continue the investigation could you post the exact switches you used to create the AOT cache?

@mcculls

Hi @mcculls - Thanks for reaching out.
And sorry — the command I wrote was wrong.
The -XX:AOT part should have been -Xlog:aot. I have updated the issue description as well.

This is the command I use to create the AOT cache:

export JDK_JAVA_OPTIONS="-javaagent:dd-java-agent.jar \
-Xmx256m \
-Xms256m \
-Dspring.context.exit=onRefresh \
-Xlog:aot \
-XX:AOTCacheOutput=/workspace/application.aot"
java -jar application.jar

Alternatively, I also create a class to terminate the application and use a method like this:

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.ApplicationContext;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Component;

@Component
@ConditionalOnProperty(prefix = "aot.training", name = "enabled", havingValue = "true")
public class AOTTrainingTerminator {
  private final Logger logger = LoggerFactory.getLogger(AOTTrainingTerminator.class);
  private final ApplicationContext context;

  public AOTTrainingTerminator(ApplicationContext context) {
    this.context = context;
  }

  @EventListener(ApplicationReadyEvent.class)
  public void terminate() {
    logger.info("Terminating the AOT training.");
    System.exit(SpringApplication.exit(context, () -> 0));
  }
}

export AOT_TRAINING_ENABLED=true
export JDK_JAVA_OPTIONS="-javaagent:dd-java-agent.jar \
-Xmx256m \
-Xms256m \
-Xlog:aot \
-XX:AOTCacheOutput=/workspace/application.aot"
java -jar application.jar

@footaku

Hi @footaku - can you post the full version string of your JVM, thanks

@mcculls

Hi @mcculls - this is the version string of the JVM:

java --version
openjdk 25 2025-09-16 LTS
OpenJDK Runtime Environment Temurin-25+36 (build 25+36-LTS)
OpenJDK 64-Bit Server VM Temurin-25+36 (build 25+36-LTS, mixed mode, sharing)

We have a similar set up and encounter the same problem. Without -XX:-AOTClassLinking we get a 60% improvement in startup resource usage, but the profiling data doesn't make it into Datadog properly (we notice the dd tags are missing). With -XX:-AOTClassLinking our improvement drops to 10%, but our traces appear correctly.

we are similarly seeing issues. In our cases its not all traces but just subset of them missing. Seems to be mainly automatically instrumented things, while manual trace usage works

Can also confirm the bug. This is preventing us from using AOT, which is unfortunate.

@mccu

@mccu any update on this?

@snuderl

@snuderl @vigenere23 can you post your JVM details?

If you're able to share a buildable example that demonstrates the issue that would really help progress this issue and ensure any fix will also apply to your setup - so far simple examples such as Spring petclinic don't exhibit this issue.

Based on the agent feedback the reason petclinic does not reproduce is because it uses synchronous MVC which partially works.

Anyway we are seeing missing stuff in atleast 3 places:

kafka traces
http
scheduled tasks in spring

I have included this patch against petclinic that shows all 3 of the issues.

petclinic-aot-reproducer.patch Note also it does not seem to be related to specific jvm distribution (I tried a few)

# 1. Clone petclinic
git clone https://github.com/spring-projects/spring-petclinic.git
cd spring-petclinic
 
# 2. Apply patch
git apply petclinic-aot-reproducer.patch
 
# 3. Download prerequisites
wget https://corretto.aws/downloads/latest/amazon-corretto-25-x64-linux-jdk.tar.gz
tar xzf amazon-corretto-25-x64-linux-jdk.tar.gz
export JAVA_HOME=$(pwd)/amazon-corretto-25.0.3.9.1-linux-x64
 
wget -O dd-java-agent.jar https://dtdg.co/latest-java-tracer
export DD_AGENT=$(pwd)/dd-java-agent.jar
 
# 4. Run
bash test-aot.sh

which gives me the following output:

=== SUMMARY ===
  classlink:                 12 failures (6 unique)
  noclasslink:                2 failures (1 unique)
  noaot:                      2 failures (1 unique)

@snuderl

thanks @snuderl!

snuderl's repro worked for me. We have this problem with synchronous MVC, it appears somehow related to jib with containerizingMode=packaged* rather than the fat jar. When trying that out the 12 failures turn into 90+ including JDBC/Hibernate/HikariCP/CoyoteAdapter/DispatcherServlet.

I'm still looking into the trace differences but wanted to mention that the Failed to add context-store field messages are expected when using AOT.

That message means the tracer wasn't able to inject a field into the class. This can happen when classes are materialized straight from the AOT cache because they bypass the usual JVM instrumentation layer. Once the class is loaded the JVM doesn't let you add a field to the class. Basically you cannot change the shape of the class, but you can change what a method does - so you can still apply entry-exit advice.

Note this shouldn't lead to missing traces because the instrumentation will automatically fall back to the global weak-map to track context - so something else is going on that does affect traces in certain circumstances.

I was able to reproduce it (with some kafka consumer / producer / apis) with my app and in a test directly in the library and after investigating I found out that the global weak-map is not necessarily cleaning all entries fast enough and therefore at some point we go beyond the 50k hard cap by second of entries and so following objects are just trashed. With AOT disabled the issue is not there since the global weak map fallback is not used at all.
I created a Gist here for reproducer: https://gist.github.com/CharlyRien/b50510ace2659c3d2d68abbd5dc34e11

@CharlyRien

Hi @CharlyRien - thanks for the insight, that would explain a lot.

I've recently been working on an improved weak-map approach - will try it with your reproducer app to see if it helps.

Fix async context loss when field injection is unavailable

@mcculls

Any update on this @mcculls?

@Meemaw

Hi @Meemaw - I now have a weak-map approach that avoids any dropped context (0% loss) while still respecting an upper bound.

The instrumentation fix is already in place but the new weak-map won't make it into 1.65.0 (too close to the scheduled release to safely get it in) - I do expect it to be part of 1.66.0 expected at the start of September.

@davidgoldtempo

TL;DR: we now have all spans collected with the AOT cache still enabled — the fix was disabling a single profiler feature: -Ddd.profiling.queueing.time.enabled=false.

I hit the same symptom (JDK 25 + AOT cache + jib containerizingMode=packaged: tracer starts, agent_error: false, but zero spans) and traced it to a different mechanism than the context-store fallback discussed above.

Reproduced with both dd-java-agent 1.65.1 and 1.66.0 — identical zero-span behaviour, so this is not a recent regression in the agent.

UnwrappingVisitor (queueing-time profiling) adds the TaskWrapper interface — a structural change the JVM rejects on retransformation. With an AOT cache the affected classes are materialized before premain, so the batch fails, and since there is no BatchAllocator configured, RedefinitionStrategy aborts the whole install. That would also explain why @davidgoldtempo saw failures jump to 90+ with jib packaged: more classes land in that single doomed batch.

I've opened #12506 which retries the failed batch without the interface change, so instrumentation survives. Queueing-time profiling keeps working too — only the task field's type resolution is lost (wrapper class instead of the real one; measured with JFR in #12506), instead of the feature being disabled outright as the flag above does.

@gibkim

Hi @gibkim - thanks for the detail, but your particular issue is unrelated to this one (to confirm I tried your patch from #12506 with the reproducer in https://gist.github.com/CharlyRien/b50510ace2659c3d2d68abbd5dc34e11 and it didn't help, neither did using the -Ddd.profiling.queueing.time.enabled=false switch)

@CharlyRien

Hi @CharlyRien @Meemaw v1.66.0 shipped with a feature to help with the loss of context under load on AOT.

To use this mode set the following environment variable:

DD_TRACE_RUNTIME_CONTEXT_MAP_PER_STORE=false

or add this JVM option:

-Ddd.trace.runtime.context.map-per-store=false

The next release will include some performance improvements for this mode and hopefully start rolling it out by default. (we'll most likely do this gradually based on the Java version being used.)

@mcculls

Thanks @mcculls, I confirm that the fix works well, we are testing it in pre-prod for now, and will deploy it soon on our java services fleet.

Thanks again.

@gibkim

Upon testing out with DD_TRACE_RUNTIME_CONTEXT_MAP_PER_STORE=false we found traces were still dropped. Notably it's only when we have profiling enabled, which I could be wrong about but I believe that was the case for us before version 1.66.

Here's an attempt to repro the problem, including a run with the flag @gibkim mentioned above. Sorry for the vibe coding, hope it is helpful. https://gist.github.com/davidgoldtempo/312b18e8790f5afe4c0913e34f96e613

@davidgoldtempo

Hi @davidgoldtempo if it's only when profiling is enabled then it will be related to #12540 (which has a different root cause to the context store issue, even though the resulting effect is the same)

Thanks for the repro

Navigation Menu

Repository navigation

Traces do not appear in Datadog Web UI when AOT Cache is enabled, #10479

Tracer Version(s)

Java Version(s)

JVM Vendor

Bug Report

Behavior

Expected Behavior

Reproduction Code

Case 1 (NG)

Case 2 (OK)

Case 3 (OK)

Environment

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Traces do not appear in Datadog Web UI when AOT Cache is enabled, #10479

Description

Tracer Version(s)

Java Version(s)

JVM Vendor

Bug Report

Behavior

Expected Behavior

Reproduction Code

Case 1 (NG)

Case 2 (OK)

Case 3 (OK)

Environment

Activity

mcculls commented on Jan 30, 2026

footaku commented on Jan 30, 2026

mcculls commented on Jan 30, 2026

footaku commented on Jan 30, 2026

davidgoldtempo commented on Mar 18, 2026

snuderl commented on Apr 21, 2026

vigenere23 commented on May 22, 2026

snuderl commented on Jun 9, 2026

mcculls commented on Jun 9, 2026

snuderl commented on Jun 11, 2026

mcculls commented on Jun 11, 2026

davidgoldtempo commented on Jun 11, 2026

mcculls commented on Jun 26, 2026

CharlyRien commented on Jun 30, 2026

mcculls commented on Jun 30, 2026

Meemaw commented on Jul 30, 2026

mcculls commented on Jul 31, 2026

gibkim commented on Sep 16, 2026

mcculls commented on Sep 16, 2026

mcculls commented on Sep 16, 2026

CharlyRien commented on Sep 16, 2026

davidgoldtempo commented on Sep 22, 2026

mcculls commented on Sep 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions