Repository navigation
JVM SIGSEGV crashes after upgrade from 1.61.1 to 1.62.0 #11373
Description
Activity
Seeing the same thing -
amazoncorretto:21-alpine(Resolves to21.0.11) and dd-trace-java latest, (resolves to1.62.0Results in a segfault after a short amount of time.Pinning corretto to
21.0.10and dd-trace-java1.60.1seems to be a working combination for us.# Dockerfile FROM amazoncorretto:21.0.10-alpine [ ... ] RUN mkdir -p /opt/datadog && \ curl -Lo /opt/datadog/dd-java-agent.jar 'https://github.com/DataDog/dd-trace-java/releases/download/v1.60.1/dd-java-agent-1.60.1.jar' [ ... ]Please, upgrade to 1.63.0 - we have a number of fixes in profiler that are targeting the concurrency issues and memory corruption. Internal testing and dogfooding is showing that the problems are gone in 1.63.0
Reacted by elue and 제이스We had been seeing the same crashes on our side, usually several dozen to hundreds per day after the update to 1.62.0. So far after the update to 1.63.0 it has drastically improved but we are not totally on 0, I'm aware of 3 restarts so far happening in different apps each with 1 during 24 hours. They all had the same reference
# A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007f04b6325324, pid=1, tid=527 # # JRE version: OpenJDK Runtime Environment (Red_Hat-21.0.9.0.10-1) (21.0.9+10) (build 21.0.9+10-LTS) # Java VM: OpenJDK 64-Bit Server VM (Red_Hat-21.0.9.0.10-1) (21.0.9+10-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64) # Problematic frame: # V [libjvm.so+0x9e0324] JfrArtifactCallbackHost<Klass const*, CompositeFunctor<Klass const*, JfrTypeWriterHost<JfrPredicatedTypeWriterImplHost<Klass const*, SerializePredicate<Klass const*>, &(write__klass(JfrCheckpointWriter*, void const*))>, 178u>, KlassArtifactRegistrator> >::do_artifact(void const*)+0x24I'll continue keeping an eye on it but so far it looks like at least a pretty drastic improvement while not yet 100% stable
Reacted by Joao Leite
Tracer Version(s)
1.62.0
Java Version(s)
25
JVM Vendor
Eclipse Adoptium / Temurin
Bug Report
After upgrading dd-trace-java from 1.61.1 to 1.62.0, the JVM started crashing with SIGSEGV (and one UNKNOWN signal) on multiple tasks.
Reverting to 1.61.1 with no other changes made the crashes stop.
Crash tracking is enabled, and we have collected several distinct crash signatures. They cluster into three groups:
SIGSEGV inside the Datadog profiler native dump path:
Profiler::updateThreadName(jvmtiEnv*, JNIEnv, _jobject, bool)
-> jvmti_Deallocate
called from JavaProfiler.dump0 via the profiling system's periodic
snapshot task (AgentTaskScheduler$PeriodicTask).
SIGSEGV in JIT inline-cache / wrong-method handling:
SharedRuntime::find_callee_info_helper
-> handle_ic_miss_helper / handle_wrong_method
Observed on multiple call sites, including:
(datadog.okhttp3...Http1Codec$FixedLengthSink.close, called from
DefaultConfigurationPoller.fetchConfiguration), seen repeatedly
(io.micrometer.core.instrument.step.StepTuple2.rollCount)
JVM-internal crashes around the code cache / GC:
on a G1 code-cache unloading worker
on a C1 CompileBroker thread installing a freshly compiled nmethod
(crashed inside concurrentHashTable.inline.hpp:684)
All crashes occur either on a Datadog-managed thread or in code installed by the agent (its shaded okhttp). The stacks above come from dd-trace-java's own crashtracking output, which was enabled at the time.
hs_err_pid*.log files are not available: this is running on AWS ECS, and the crashing tasks were recycled before we could retrieve them.
Workaround: pinning back to 1.61.1 removes the crashes entirely.
logs.txt
Expected Behavior
Upgrading dd-trace-java from 1.61.1 to 1.62.0 should not cause JVM crashes.
Reproduction Code
No response