Skip to content

Cleanup of zstd dependency and jps functionality #9010

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 19, 2025
Merged

Cleanup of zstd dependency and jps functionality #9010

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
86ff6ed
Make the zstd dependency more lightweight
jbachorik Jun 19, 2025
2587735
Simplify the JVMs pids collection
jbachorik Jun 19, 2025
48ab511
Merge branch 'master' into jb/zstd_improvement
jbachorik Jun 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 0 additions & 21 deletions dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/Agent.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
import static datadog.trace.api.ConfigDefaults.DEFAULT_STARTUP_LOGS_ENABLED;
import static datadog.trace.api.Platform.isJavaVersionAtLeast;
import static datadog.trace.api.Platform.isOracleJDK8;
import static datadog.trace.api.telemetry.LogCollector.SEND_TELEMETRY;
import static datadog.trace.bootstrap.Library.WILDFLY;
import static datadog.trace.bootstrap.Library.detectLibraries;
import static datadog.trace.util.AgentThreadFactory.AgentThread.JMX_STARTUP;
Expand Down Expand Up @@ -45,7 +44,6 @@
import datadog.trace.util.AgentTaskScheduler;
import datadog.trace.util.AgentThreadFactory.AgentThread;
import datadog.trace.util.throwable.FatalAgentMisconfigurationError;
import de.thetaphi.forbiddenapis.SuppressForbidden;
import java.lang.instrument.Instrumentation;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
Expand Down Expand Up @@ -291,8 +289,6 @@ public static void start(
codeOriginEnabled = isFeatureEnabled(AgentFeature.CODE_ORIGIN);
agentlessLogSubmissionEnabled = isFeatureEnabled(AgentFeature.AGENTLESS_LOG_SUBMISSION);

patchJPSAccess(inst);

if (profilingEnabled) {
if (!isOracleJDK8()) {
// Profiling agent startup code is written in a way to allow `startProfilingAgent` be called
Expand Down Expand Up @@ -422,23 +418,6 @@ private static void injectAgentArgsConfig(String agentArgs) {
}
}

@SuppressForbidden
public static void patchJPSAccess(Instrumentation inst) {
if (Platform.isJavaVersionAtLeast(9)) {
// Unclear if supported for J9, may need to revisit
try {
Class.forName("datadog.trace.util.JPMSJPSAccess")
.getMethod("patchModuleAccess", Instrumentation.class)
.invoke(null, inst);
} catch (Exception e) {
log.debug(
SEND_TELEMETRY,
"Failed to patch module access for jvmstat and Java version "
+ Platform.getRuntimeVersion());
}
}
}

public static void shutdown(final boolean sync) {
StaticEventLogger.end("Agent");
StaticEventLogger.stop();
Expand Down
10 changes: 10 additions & 0 deletions dd-java-agent/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,16 @@ ext.generalShadowJarConfig = {
exclude '**/com/kenai/jffi/Init.class'
relocate('com.kenai.jffi.Init', 'com.kenai.jffi.PatchInit')

// Minimize and relocate the airlift compressor dependency for ZSTD
exclude '**/io/airlift/compress/bzip2/**'
exclude '**/io/airlift/compress/deflate/**'
exclude '**/io/airlift/compress/gzip/**'
exclude '**/io/airlift/compress/hadoop/**'
exclude '**/io/airlift/compress/lz4/**'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Out of curiosity is there a reason to prefer lz4-java over aircompressor lz4 ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really, it's just that we know lz4-java works and it didn't feel good to do the change as a part of last minute cleanup PR :)
But definitely we should followup on this - it does not make sense pulling in two different libraries when we can handle the supported compression formats by a single one.

exclude '**/io/airlift/compress/lzo/**'
exclude '**/io/airlift/compress/snappy/**'
relocate 'io.airlift', 'datadog.io.airlift'

final String projectName = "${project.name}"

// Prevents conflict with other instances, but doesn't relocate instrumentation
Expand Down
4 changes: 3 additions & 1 deletion dd-smoke-tests/profiling-integration-tests/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,16 @@ dependencies {
implementation project(':dd-trace-api')
api project(':dd-trace-ot')
implementation 'org.apache.commons:commons-math3:3.6.1'
implementation 'org.lz4:lz4-java:1.8.0'
implementation libs.lz4
implementation libs.aircompressor
implementation 'org.xerial.snappy:snappy-java:1.1.8.4'

testImplementation project(':dd-smoke-tests')
testImplementation project(':dd-java-agent:agent-profiling:profiling-testing')
testImplementation libs.bundles.junit5
testImplementation libs.bundles.mockito
testImplementation libs.bundles.jmc
testImplementation libs.aircompressor
testImplementation(group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.9.10')
}

Expand Down
38 changes: 24 additions & 14 deletions ...g-integration-tests/src/test/java/datadog/smoketest/JFRBasedProfilingIntegrationTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,11 @@
import datadog.trace.api.Platform;
import datadog.trace.api.config.ProfilingConfig;
import delight.fileupload.FileUpload;
import io.airlift.compress.zstd.ZstdInputStream;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.net.InetAddress;
import java.nio.file.Files;
import java.nio.file.Path;
Expand Down Expand Up @@ -150,7 +152,7 @@ void teardown() throws Exception {
}

@Test
@DisplayName("Test continuous recording - no jmx delay, no jmethodid cache")
@DisplayName("Test continuous recording - no jmx delay, default compression")
public void testContinuousRecording_no_jmx_delay(final TestInfo testInfo) throws Exception {
testWithRetry(
() ->
Expand All @@ -161,7 +163,7 @@ public void testContinuousRecording_no_jmx_delay(final TestInfo testInfo) throws
}

@Test
@DisplayName("Test continuous recording - no jmx delay, jmethodid cache")
@DisplayName("Test continuous recording - no jmx delay, zstd compression")
public void testContinuousRecording_no_jmx_delay_jmethodid_cache(final TestInfo testInfo)
throws Exception {
testWithRetry(
Expand All @@ -173,7 +175,7 @@ public void testContinuousRecording_no_jmx_delay_jmethodid_cache(final TestInfo
}

@Test
@DisplayName("Test continuous recording - 1 sec jmx delay, no jmethodid cache")
@DisplayName("Test continuous recording - 1 sec jmx delay, default compression")
public void testContinuousRecording(final TestInfo testInfo) throws Exception {
testWithRetry(
() ->
Expand All @@ -184,7 +186,7 @@ public void testContinuousRecording(final TestInfo testInfo) throws Exception {
}

@Test
@DisplayName("Test continuous recording - 1 sec jmx delay, jmethodid cache")
@DisplayName("Test continuous recording - 1 sec jmx delay, zstd compression")
public void testContinuousRecording_jmethodid_cache(final TestInfo testInfo) throws Exception {
testWithRetry(
() ->
Expand All @@ -198,7 +200,7 @@ private void testContinuousRecording(
final int jmxFetchDelay,
final boolean endpointCollectionEnabled,
final boolean asyncProfilerEnabled,
final boolean jmethodIdCacheEnabled)
final boolean withZstd)
throws Exception {
final ObjectMapper mapper = new ObjectMapper();
try {
Expand All @@ -207,7 +209,7 @@ private void testContinuousRecording(
jmxFetchDelay,
endpointCollectionEnabled,
asyncProfilerEnabled,
jmethodIdCacheEnabled,
withZstd,
logFilePath)
.start();

Expand Down Expand Up @@ -261,7 +263,11 @@ private void testContinuousRecording(
assertEquals(InetAddress.getLocalHost().getHostName(), requestTags.get("host"));

assertFalse(logHasErrors(logFilePath));
IItemCollection events = JfrLoaderToolkit.loadEvents(new ByteArrayInputStream(rawJfr.get()));
InputStream eventStream = new ByteArrayInputStream(rawJfr.get());
if (withZstd) {
eventStream = new ZstdInputStream(eventStream);
}
IItemCollection events = JfrLoaderToolkit.loadEvents(eventStream);
assertTrue(events.hasItems());
Pair<Instant, Instant> rangeStartAndEnd = getRangeStartAndEnd(events);
// This nano-second compensates for the added nano second in
Expand Down Expand Up @@ -308,7 +314,11 @@ private void testContinuousRecording(
period > 0 && period <= upperLimit,
() -> "Upload period = " + period + "ms, expected (0, " + upperLimit + "]ms");

events = JfrLoaderToolkit.loadEvents(new ByteArrayInputStream(rawJfr.get()));
eventStream = new ByteArrayInputStream(rawJfr.get());
if (withZstd) {
eventStream = new ZstdInputStream(eventStream);
}
events = JfrLoaderToolkit.loadEvents(eventStream);
assertTrue(events.hasItems());
verifyDatadogEventsNotCorrupt(events);
rangeStartAndEnd = getRangeStartAndEnd(events);
Expand Down Expand Up @@ -689,7 +699,7 @@ private ProcessBuilder createDefaultProcessBuilder(
final int jmxFetchDelay,
final boolean endpointCollectionEnabled,
final boolean asyncProfilerEnabled,
final boolean jmethodIdCacheEnabled,
final boolean withZstd,
final Path logFilePath) {
return createProcessBuilder(
VALID_API_KEY,
Expand All @@ -698,7 +708,7 @@ private ProcessBuilder createDefaultProcessBuilder(
PROFILING_UPLOAD_PERIOD_SECONDS,
endpointCollectionEnabled,
asyncProfilerEnabled,
jmethodIdCacheEnabled,
withZstd,
0,
logFilePath);
}
Expand All @@ -710,7 +720,7 @@ private ProcessBuilder createProcessBuilder(
final int profilingUploadPeriodSecs,
final boolean endpointCollectionEnabled,
final boolean asyncProfilerEnabled,
final boolean jmethodIdCacheEnabled,
final boolean withZstd,
final int exitDelay,
final Path logFilePath) {
return createProcessBuilder(
Expand All @@ -722,7 +732,7 @@ private ProcessBuilder createProcessBuilder(
profilingUploadPeriodSecs,
endpointCollectionEnabled,
asyncProfilerEnabled,
jmethodIdCacheEnabled,
withZstd,
exitDelay,
logFilePath);
}
Expand All @@ -736,7 +746,7 @@ private static ProcessBuilder createProcessBuilder(
final int profilingUploadPeriodSecs,
final boolean endpointCollectionEnabled,
final boolean asyncProfilerEnabled,
final boolean jmethodIdCacheEnabled,
final boolean withZstd,
final int exitDelay,
final Path logFilePath) {
final String templateOverride =
Expand Down Expand Up @@ -770,7 +780,7 @@ private static ProcessBuilder createProcessBuilder(
"-Ddd.profiling.debug.dump_path=/tmp/dd-profiler",
"-Ddd.profiling.queueing.time.enabled=true",
"-Ddd.profiling.queueing.time.threshold.millis=0",
"-Ddd.profiling.experimental.jmethodid_cache.enabled=" + jmethodIdCacheEnabled,
"-Ddd.profiling.debug.upload.compression=" + (withZstd ? "zstd" : "on"),
"-Ddatadog.slf4j.simpleLogger.defaultLogLevel=debug",
"-Ddd.profiling.context.attributes=foo,bar",
"-Dorg.slf4j.simpleLogger.defaultLogLevel=debug",
Expand Down
4 changes: 3 additions & 1 deletion gradle/dependencies.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ final class CachedData {
exclude(dependency('com.datadoghq.okio:okio'))
exclude(dependency('com.squareup.okio:okio'))
exclude(dependency('org.lz4:lz4-java'))
exclude(dependency('io.airlift:aircompressor'))

// dogstatsd and its transitives
exclude(dependency('com.datadoghq:java-dogstatsd-client'))
Expand Down Expand Up @@ -67,7 +68,8 @@ CachedData.deps.shared = [
libs.jnr.unixsocket,
libs.moshi,
libs.jctools,
libs.lz4
libs.lz4,
libs.aircompressor
]

ext {
Expand Down
40 changes: 0 additions & 40 deletions internal-api/internal-api-9/src/main/java/datadog/trace/util/JPMSJPSAccess.java
View file
Open in desktop

This file was deleted.

21 changes: 0 additions & 21 deletions internal-api/internal-api-9/src/test/groovy/datadog/trace/util/PidHelperTest.groovy
View file
Open in desktop

This file was deleted.

25 changes: 0 additions & 25 deletions internal-api/src/main/java/datadog/trace/util/JPSUtils.java
View file
Open in desktop

This file was deleted.

Loading