Skip to content

Virtual threads silently never run their body under dd-java-agent with a JDK 26 AOT cache #12196

Description

@snuderl

Tracer Version(s)

1.63.0

Java Version(s)

Corretto-26.0.1.8.1

JVM Vendor

Amazon Corretto

Bug Report

On JDK 26 with an AOT cache (-XX:AOTMode=on, JEP 483/515), attaching dd-java-agent makes every virtual thread terminate without ever executing its Runnable. Thread.ofVirtual().start(r) returns, the thread goes to TERMINATED, and r is never invoked. No exception is thrown, nothing is logged, and platform threads are unaffected — the work is silently discarded.

Anything that awaits virtual-thread work therefore hangs forever. In our case a Spring Boot service never finished startup (its Kafka producer warmup runs on Executors.newVirtualThreadPerTaskExecutor()), so all pods crashlooped in production while the identical binary without the AOT cache was fine.

The trigger is narrow: the AOT cache must have been recorded by a run that (a) had --enable-final-field-mutation=ALL-UNNAMED (JEP 500) set and (b) actually created a virtual thread. The flag is irrelevant at run time — only what the cache was recorded with matters, which makes this very hard to attribute: the run that breaks does not have to mention the flag at all.

-Ddd.integration.virtual-thread.enabled=false avoids it, which points at datadog.trace.instrumentation.java.lang.jdk21.VirtualThreadInstrumentation.

Reproducer
VirtualThreadAotRepro.java + run.sh (attached). It reports whether the body of a platform thread, a virtual thread, and a task submitted to newVirtualThreadPerTaskExecutor() actually executed:

JAVA_HOME=/path/to/jdk26 DD_AGENT_JAR=/path/to/dd-java-agent.jar ./run.sh
Minimal manual version:

1. record an AOT cache with the JEP 500 flag, from a run that uses a virtual thread

java -XX:AOTMode=record -XX:AOTCacheOutput=app.aot \
     --enable-final-field-mutation=ALL-UNNAMED \
     -javaagent:dd-java-agent.jar -cp repro.jar VirtualThreadAotRepro

2. run with that cache and the agent — the virtual thread bodies never run

java -XX:AOTMode=on -XX:AOTCache=app.aot \
     -javaagent:dd-java-agent.jar -cp repro.jar VirtualThreadAotRepro
Observed vs expected
Expected: three body ran=true lines. Observed in the broken combination:

platform thread : body ran=true alive=false state=TERMINATED
virtual thread : body ran=false alive=false state=TERMINATED <-- Runnable never invoked
vthread executor: body ran=false (latch released=false) <-- task never runs, await times out
Full matrix from run.sh
The agent is attached during recording in every row below, as it is in production (a cache recorded without -javaagent cannot be loaded by a run that adds one — module graph mismatch).

AOT cache recorded with run with result
(no cache) agent + flag PASS
(no cache) no agent, no flag PASS
flag, virtual threads used agent + flag FAIL
flag, virtual threads used agent, no flag FAIL
flag, virtual threads used no agent, flag PASS
flag, virtual threads used no agent, no flag PASS
no flag, virtual threads used agent PASS
flag, virtual threads NOT used agent + flag PASS
So all three of these are required: the JEP 500 flag at record time, a virtual thread created during the recording run, and the agent attached at run time.

Narrowing inside the agent
Run against the broken cache:

additional property result
-Ddd.trace.enabled=false FAIL
-Ddd.trace.runtime.context.field.injection=false FAIL
-Ddd.integration.java_concurrent.enabled=false FAIL
-Ddd.integrations.enabled=false PASS
-Ddd.integration.virtual-thread.enabled=false PASS

Disabling tracing is not enough — instrumentation is what matters, and specifically the virtual-thread integration (java/lang/jdk21/VirtualThreadInstrumentation, whose Construct, Mount, Unmount and AfterDone advice weaves java.lang.VirtualThread). Our reading, which we have not confirmed against the JVM internals, is that when VirtualThread is AOT-linked from a cache recorded under relaxed final-field-mutation semantics, that advice's effect on the thread's task/state field is lost, so the thread has nothing to run. Whatever the precise cause, retransforming an AOT-linked VirtualThread should not be able to silently drop the task.

Expected Behavior

Virtual threads should work :)

Reproduction Code

No response

Activity

  1. self-assigned this
    on Aug 17, 2026
  2. mcculls commented on Aug 17, 2026

    @mcculls
    Contributor

    Hi @snuderl - can you attach VirtualThreadAotRepro.java and your run.sh - thanks

  3. snuderl commented on Aug 20, 2026

    @snuderl
    Author

    Bad agent bad.

    Anyway here are the files

    vt-aot-repro.tar.gz

  4. mcculls commented on Sep 21, 2026

    @mcculls
    Contributor

    Hi @snuderl - I noticed a small inconsistency in your run.sh script - the following line:

    run "agent, no flag      <-- also broken"            -XX:AOTMode=on -XX:AOTCache=flag-vthreads.aot ${AGENT_OPTS}
    

    should really be:

    run "agent, no flag      <-- also broken"            -XX:AOTMode=on -XX:AOTCache=noflag-vthreads.aot ${AGENT_OPTS}
    

    i.e. it should use the noflag-vthreads.aot recording which was made without the --enable-final-field-mutation=ALL-UNNAMED JVM flag.

    When I make that change to the script this particular scenarios PASSES - indicating that this issue is related to using the --enable-final-field-mutation=ALL-UNNAMED JVM flag with AOT.

    Digging a bit further it appears that using --enable-final-field-mutation=ALL-UNNAMED disables the full module graph and optimized module handling features of AOT/CDS. This in turn leads to an issue restoring classes from the AOT cache which then appears to affect retransformation of those classes.

    The workaround at the moment is to avoid using --enable-final-field-mutation=ALL-UNNAMED with AOT if possible.

  5. mcculls commented on Sep 21, 2026

    @mcculls
    Contributor

    BTW Java 28 EA fails to create an AOT cache with --enable-final-field-mutation=ALL-UNNAMED without any agent involved:

    java --enable-final-field-mutation=ALL-UNNAMED -XX:AOTMode=record -XX:AOTCacheOutput=flag-java28ea.aot -cp repro.jar VirtualThreadAotRepro
    
    platform thread : body ran=true alive=false state=TERMINATED
    virtual thread  : body ran=true alive=false state=TERMINATED
    vthread executor: body ran=true (latch released=true)
    RESULT: PASS
    Temporary AOTConfiguration recorded: flag-java28ea.aot.config
    Launching child process /Library/Java/JavaVirtualMachines/jdk-28.jdk/Contents/Home/bin/java to assemble AOT cache flag-java28ea.aot using configuration flag-java28ea.aot.config
    Picked up JAVA_TOOL_OPTIONS: -Djava.class.path=repro.jar --enable-final-field-mutation=ALL-UNNAMED -XX:AOTCacheOutput=flag-java28ea.aot -XX:AOTConfiguration=flag-java28ea.aot.config -XX:AOTMode=create
    [0.007s][error][aot] An error has occurred while writing the AOT cache.
    [0.007s][error][aot] AOT class linking was enabled in training run but has been disabled due to incompatible module options
    [0.336s][error][aot] Child process failed; status = 1
    

    The AOT cache is successfully created once I remove that flag:

    java -XX:AOTCacheOutput=noflag-java28ea.aot -cp repro.jar VirtualThreadAotRepro
    
    platform thread : body ran=true alive=false state=TERMINATED
    virtual thread  : body ran=true alive=false state=TERMINATED
    vthread executor: body ran=true (latch released=true)
    RESULT: PASS
    Temporary AOTConfiguration recorded: noflag-java28ea.aot.config
    Launching child process /Library/Java/JavaVirtualMachines/jdk-28.jdk/Contents/Home/bin/java to assemble AOT cache noflag-java28ea.aot using configuration noflag-java28ea.aot.config
    Picked up JAVA_TOOL_OPTIONS: -Djava.class.path=repro.jar -XX:AOTCacheOutput=noflag-java28ea.aot -XX:AOTConfiguration=noflag-java28ea.aot.config -XX:AOTMode=create
    Reading AOTConfiguration noflag-java28ea.aot.config and writing AOTCache noflag-java28ea.aot
    AOTCache creation is complete: noflag-java28ea.aot 13156352 bytes
    Removed temporary AOT configuration file noflag-java28ea.aot.config
    

    This suggests an underlying JVM issue between that flag and AOT

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions