Avoid overlapping outputs for indexing tasks #9725
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Emit index in proper generated source location and use the location as source dir. * Deduplicate code.
… overlapping output * Expand the included agent jars in another location buildDir/included, this location contributes to the resources. * Emit index in separate location, modified index to receive input folder and output folder, this location contributes to the resources. * Fix indexer logging.
bric3
changed the title
bric3
added
tag: no release notes
Contributor
Author
|
Another idea to improve the gradleiness, use configurations: Pseudo code in Kotlin DSL when using configurationsval includedAgentDir = layout.buildDirectory.dir("generated/included")
val agentModules by configurations.registering {
isCanBeResolved = true
isCanBeConsumed = false
attributes {
attribute(Usage.USAGE_ATTRIBUTE, objects.named<Usage>("agent-module"))
attribute(Usage.USAGE_ATTRIBUTE, objects.named<Usage>(Usage::class.java, Usage.JAVA_RUNTIME))
attribute(Category.CATEGORY_ATTRIBUTE, objects.named<Category>(Category.LIBRARY))
attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, objects.named<LibraryElements>(LibraryElements.JAR))
}
}
// Declare module dependencies with metadata
data class AgentModule(
val project: String,
val targetDir: String,
val specialHandling: ((CopySpec) -> Unit)? = null
)
val moduleList = listOf(
AgentModule(":dd-java-agent:instrumentation", "inst") { spec ->
spec.duplicatesStrategy = DuplicatesStrategy.EXCLUDE
spec.eachFile {
if (path.contains("META-INF/versions/9/net/bytebuddy")) {
path = path.replace("META-INF/versions/9/", "")
}
}
},
AgentModule(":dd-java-agent:agent-jmxfetch", "metrics"),
AgentModule(":dd-java-agent:agent-profiling", "profiling"),
AgentModule(":dd-java-agent:appsec", "appsec"),
AgentModule(":dd-java-agent:agent-aiguard", "aiguard"),
AgentModule(":dd-java-agent:agent-iast", "iast"),
AgentModule(":dd-java-agent:agent-debugger", "debugger"),
AgentModule(":dd-java-agent:agent-ci-visibility", "ci-visibility"),
AgentModule(":dd-java-agent:agent-llmobs", "llm-obs"),
AgentModule(":dd-java-agent:agent-logs-intake", "logs-intake"),
AgentModule(":dd-java-agent:cws-tls", "cws-tls")
)
dependencies {
moduleList.forEach { module ->
agentModules(project(module.project))
}
}
// Single task to expand all agent modules
val expandAllAgentModules = tasks.register<Sync>("expandAllAgentModules") {
group = LifecycleBasePlugin.BUILD_GROUP
description = "Expand all agent shadow jars into subdirectories"
into(includedAgentDir)
val opentracingFound = AtomicBoolean()
moduleList.forEach { module ->
val moduleProject = project(module.project)
val shadowJarTask = moduleProject.tasks.named<ShadowJar>("shadowJar")
dependsOn(shadowJarTask)
from(zipTree(shadowJarTask.map { it.archiveFile })) {
into(module.targetDir)
rename("(^.*)\\.class$", "$1.classdata")
rename("^LICENSE$", "LICENSE.renamed")
eachFile {
if (path.contains("opentracing") && name.contains("Format\$Builtin")) {
opentracingFound.set(true)
}
}
module.specialHandling?.invoke(this)
}
}
doLast {
if (opentracingFound.get()) {
throw GradleException("OpenTracing direct dependency found!")
}
}
}
val includedJarFileTree = fileTree(includedAgentDir) {
builtBy(expandAllAgentModules)
}
// Generate agent jar index
val generateAgentJarIndex = tasks.register<JavaExec>("generateAgentJarIndex") {
val destinationDir = layout.buildDirectory.dir("generated/$name")
group = LifecycleBasePlugin.BUILD_GROUP
description = "Generate dd-java-agent.index"
mainClass.set("datadog.trace.bootstrap.AgentJarIndex\$IndexGenerator")
inputs.files(includedJarFileTree)
inputs.files(classpath)
outputs.dir(destinationDir)
classpath = objects.fileCollection().from(
configurations.named("shadowInclude"),
configurations.named("slf4j-simple")
)
argumentProviders.add(CommandLineArgumentProvider {
listOf(
includedAgentDir.get().asFile.path,
destinationDir.get().asFile.path
)
})
}
sourceSets.main {
resources.srcDir(expandAllAgentModules)
} |
bric3
commented
Oct 10, 2025
dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/AgentJarIndex.java
Show resolved
Hide resolved
Comment on lines
-55
to
-56
| // Include AgentPreCheck compiled with Java 6. | ||
| from sourceSets.main_java6.output |
Contributor
Author
There was a problem hiding this comment.
note: This was incorrectly added to all "included" sub jars, while it should only be present at the root. This was moved to the main shadowJar task.
|
🎯 Code Coverage 🔗 Commit SHA: b59e05e | Docs | Was this helpful? Give us feedback! |
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 6 unstable metrics. Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.55.0-SNAPSHOT~b59e05e66d, baseline=1.55.0-SNAPSHOT~85d85805f6
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.014 s) : 0, 1014001
Total [baseline] (8.644 s) : 0, 8643756
Agent [candidate] (1.025 s) : 0, 1024603
Total [candidate] (8.657 s) : 0, 8656961
section iast
Agent [baseline] (1.15 s) : 0, 1150053
Total [baseline] (9.28 s) : 0, 9280055
Agent [candidate] (1.149 s) : 0, 1149025
Total [candidate] (9.261 s) : 0, 9261069
gantt
title insecure-bank - break down per module: candidate=1.55.0-SNAPSHOT~b59e05e66d, baseline=1.55.0-SNAPSHOT~85d85805f6
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.467 ms) : 0, 1467
crashtracking [candidate] (1.484 ms) : 0, 1484
BytebuddyAgent [baseline] (692.377 ms) : 0, 692377
BytebuddyAgent [candidate] (698.372 ms) : 0, 698372
GlobalTracer [baseline] (240.728 ms) : 0, 240728
GlobalTracer [candidate] (243.351 ms) : 0, 243351
AppSec [baseline] (32.446 ms) : 0, 32446
AppSec [candidate] (32.971 ms) : 0, 32971
Debugger [baseline] (6.404 ms) : 0, 6404
Debugger [candidate] (6.524 ms) : 0, 6524
Remote Config [baseline] (699.106 µs) : 0, 699
Remote Config [candidate] (709.273 µs) : 0, 709
Telemetry [baseline] (9.171 ms) : 0, 9171
Telemetry [candidate] (9.23 ms) : 0, 9230
Flare Poller [baseline] (9.602 ms) : 0, 9602
Flare Poller [candidate] (10.781 ms) : 0, 10781
section iast
crashtracking [baseline] (1.483 ms) : 0, 1483
crashtracking [candidate] (1.485 ms) : 0, 1485
BytebuddyAgent [baseline] (815.378 ms) : 0, 815378
BytebuddyAgent [candidate] (813.8 ms) : 0, 813800
GlobalTracer [baseline] (230.565 ms) : 0, 230565
GlobalTracer [candidate] (231.012 ms) : 0, 231012
AppSec [baseline] (35.137 ms) : 0, 35137
AppSec [candidate] (35.055 ms) : 0, 35055
Debugger [baseline] (6.151 ms) : 0, 6151
Debugger [candidate] (6.097 ms) : 0, 6097
Remote Config [baseline] (612.605 µs) : 0, 613
Remote Config [candidate] (612.332 µs) : 0, 612
Telemetry [baseline] (8.608 ms) : 0, 8608
Telemetry [candidate] (8.606 ms) : 0, 8606
Flare Poller [baseline] (4.251 ms) : 0, 4251
Flare Poller [candidate] (4.282 ms) : 0, 4282
IAST [baseline] (26.387 ms) : 0, 26387
IAST [candidate] (26.586 ms) : 0, 26586
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.55.0-SNAPSHOT~b59e05e66d, baseline=1.55.0-SNAPSHOT~85d85805f6
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.028 s) : 0, 1028319
Total [baseline] (10.665 s) : 0, 10665146
Agent [candidate] (1.022 s) : 0, 1022030
Total [candidate] (10.758 s) : 0, 10758350
section appsec
Agent [baseline] (1.201 s) : 0, 1201485
Total [baseline] (10.99 s) : 0, 10989925
Agent [candidate] (1.196 s) : 0, 1195880
Total [candidate] (11.034 s) : 0, 11034255
section iast
Agent [baseline] (1.151 s) : 0, 1150813
Total [baseline] (11.0 s) : 0, 11000149
Agent [candidate] (1.157 s) : 0, 1156529
Total [candidate] (11.069 s) : 0, 11068852
section profiling
Agent [baseline] (1.168 s) : 0, 1167942
Total [baseline] (11.029 s) : 0, 11028752
Agent [candidate] (1.162 s) : 0, 1161897
Total [candidate] (11.002 s) : 0, 11002296
gantt
title petclinic - break down per module: candidate=1.55.0-SNAPSHOT~b59e05e66d, baseline=1.55.0-SNAPSHOT~85d85805f6
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.487 ms) : 0, 1487
crashtracking [candidate] (1.463 ms) : 0, 1463
BytebuddyAgent [baseline] (702.708 ms) : 0, 702708
BytebuddyAgent [candidate] (695.861 ms) : 0, 695861
GlobalTracer [baseline] (243.097 ms) : 0, 243097
GlobalTracer [candidate] (242.62 ms) : 0, 242620
AppSec [baseline] (32.776 ms) : 0, 32776
AppSec [candidate] (32.704 ms) : 0, 32704
Debugger [baseline] (6.475 ms) : 0, 6475
Debugger [candidate] (6.494 ms) : 0, 6494
Remote Config [baseline] (696.694 µs) : 0, 697
Remote Config [candidate] (691.222 µs) : 0, 691
Telemetry [baseline] (9.343 ms) : 0, 9343
Telemetry [candidate] (9.449 ms) : 0, 9449
Flare Poller [baseline] (10.322 ms) : 0, 10322
Flare Poller [candidate] (11.676 ms) : 0, 11676
section appsec
crashtracking [baseline] (1.476 ms) : 0, 1476
crashtracking [candidate] (1.452 ms) : 0, 1452
BytebuddyAgent [baseline] (723.616 ms) : 0, 723616
BytebuddyAgent [candidate] (718.841 ms) : 0, 718841
GlobalTracer [baseline] (235.91 ms) : 0, 235910
GlobalTracer [candidate] (235.602 ms) : 0, 235602
AppSec [baseline] (174.91 ms) : 0, 174910
AppSec [candidate] (174.881 ms) : 0, 174881
Debugger [baseline] (6.181 ms) : 0, 6181
Debugger [candidate] (6.106 ms) : 0, 6106
Remote Config [baseline] (631.047 µs) : 0, 631
Remote Config [candidate] (619.44 µs) : 0, 619
Telemetry [baseline] (8.553 ms) : 0, 8553
Telemetry [candidate] (8.421 ms) : 0, 8421
Flare Poller [baseline] (3.959 ms) : 0, 3959
Flare Poller [candidate] (3.956 ms) : 0, 3956
IAST [baseline] (24.984 ms) : 0, 24984
IAST [candidate] (24.97 ms) : 0, 24970
section iast
crashtracking [baseline] (1.459 ms) : 0, 1459
crashtracking [candidate] (1.461 ms) : 0, 1461
BytebuddyAgent [baseline] (815.437 ms) : 0, 815437
BytebuddyAgent [candidate] (819.535 ms) : 0, 819535
GlobalTracer [baseline] (231.167 ms) : 0, 231167
GlobalTracer [candidate] (232.083 ms) : 0, 232083
AppSec [baseline] (35.058 ms) : 0, 35058
AppSec [candidate] (35.316 ms) : 0, 35316
Debugger [baseline] (6.137 ms) : 0, 6137
Debugger [candidate] (6.228 ms) : 0, 6228
Remote Config [baseline] (625.423 µs) : 0, 625
Remote Config [candidate] (622.804 µs) : 0, 623
Telemetry [baseline] (8.685 ms) : 0, 8685
Telemetry [candidate] (8.726 ms) : 0, 8726
Flare Poller [baseline] (4.192 ms) : 0, 4192
Flare Poller [candidate] (4.276 ms) : 0, 4276
IAST [baseline] (26.47 ms) : 0, 26470
IAST [candidate] (26.686 ms) : 0, 26686
section profiling
crashtracking [baseline] (1.424 ms) : 0, 1424
crashtracking [candidate] (1.434 ms) : 0, 1434
BytebuddyAgent [baseline] (723.58 ms) : 0, 723580
BytebuddyAgent [candidate] (721.288 ms) : 0, 721288
GlobalTracer [baseline] (219.565 ms) : 0, 219565
GlobalTracer [candidate] (217.655 ms) : 0, 217655
AppSec [baseline] (32.678 ms) : 0, 32678
AppSec [candidate] (32.318 ms) : 0, 32318
Debugger [baseline] (7.329 ms) : 0, 7329
Debugger [candidate] (7.278 ms) : 0, 7278
Remote Config [baseline] (873.635 µs) : 0, 874
Remote Config [candidate] (788.163 µs) : 0, 788
Telemetry [baseline] (15.46 ms) : 0, 15460
Telemetry [candidate] (15.317 ms) : 0, 15317
Flare Poller [baseline] (4.153 ms) : 0, 4153
Flare Poller [candidate] (4.182 ms) : 0, 4182
ProfilingAgent [baseline] (109.741 ms) : 0, 109741
ProfilingAgent [candidate] (108.608 ms) : 0, 108608
Profiling [baseline] (110.746 ms) : 0, 110746
Profiling [candidate] (109.462 ms) : 0, 109462
LoadParameters
See matching parameters
SummaryFound 2 performance improvements and 1 performance regressions! Performance is the same for 9 metrics, 12 unstable metrics.
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.55.0-SNAPSHOT~b59e05e66d, baseline=1.55.0-SNAPSHOT~85d85805f6
dateFormat X
axisFormat %s
section baseline
no_agent (36.817 ms) : 36519, 37115
. : milestone, 36817,
appsec (47.662 ms) : 47258, 48066
. : milestone, 47662,
code_origins (45.113 ms) : 44724, 45503
. : milestone, 45113,
iast (45.978 ms) : 45590, 46365
. : milestone, 45978,
profiling (48.555 ms) : 48101, 49009
. : milestone, 48555,
tracing (45.729 ms) : 45342, 46116
. : milestone, 45729,
section candidate
no_agent (36.646 ms) : 36345, 36946
. : milestone, 36646,
appsec (47.887 ms) : 47461, 48313
. : milestone, 47887,
code_origins (43.056 ms) : 42692, 43420
. : milestone, 43056,
iast (44.79 ms) : 44415, 45165
. : milestone, 44790,
profiling (47.475 ms) : 47047, 47903
. : milestone, 47475,
tracing (44.964 ms) : 44601, 45327
. : milestone, 44964,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.55.0-SNAPSHOT~b59e05e66d, baseline=1.55.0-SNAPSHOT~85d85805f6
dateFormat X
axisFormat %s
section baseline
no_agent (4.446 ms) : 4390, 4501
. : milestone, 4446,
iast (9.469 ms) : 9314, 9623
. : milestone, 9469,
iast_FULL (14.17 ms) : 13885, 14455
. : milestone, 14170,
iast_GLOBAL (10.55 ms) : 10348, 10752
. : milestone, 10550,
profiling (9.263 ms) : 9116, 9411
. : milestone, 9263,
tracing (7.845 ms) : 7730, 7961
. : milestone, 7845,
section candidate
no_agent (4.476 ms) : 4425, 4527
. : milestone, 4476,
iast (9.882 ms) : 9716, 10048
. : milestone, 9882,
iast_FULL (13.708 ms) : 13441, 13976
. : milestone, 13708,
iast_GLOBAL (10.52 ms) : 10333, 10708
. : milestone, 10520,
profiling (8.729 ms) : 8589, 8868
. : milestone, 8729,
tracing (7.827 ms) : 7716, 7939
. : milestone, 7827,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.55.0-SNAPSHOT~b59e05e66d, baseline=1.55.0-SNAPSHOT~85d85805f6
dateFormat X
axisFormat %s
section baseline
no_agent (1.474 ms) : 1462, 1485
. : milestone, 1474,
appsec (3.722 ms) : 3503, 3941
. : milestone, 3722,
iast (2.204 ms) : 2141, 2267
. : milestone, 2204,
iast_GLOBAL (2.241 ms) : 2178, 2305
. : milestone, 2241,
profiling (2.044 ms) : 1993, 2094
. : milestone, 2044,
tracing (2.012 ms) : 1963, 2061
. : milestone, 2012,
section candidate
no_agent (1.474 ms) : 1462, 1485
. : milestone, 1474,
appsec (3.692 ms) : 3475, 3909
. : milestone, 3692,
iast (2.201 ms) : 2137, 2264
. : milestone, 2201,
iast_GLOBAL (2.253 ms) : 2190, 2317
. : milestone, 2253,
profiling (2.046 ms) : 1995, 2097
. : milestone, 2046,
tracing (2.032 ms) : 1982, 2081
. : milestone, 2032,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.55.0-SNAPSHOT~b59e05e66d, baseline=1.55.0-SNAPSHOT~85d85805f6
dateFormat X
axisFormat %s
section baseline
no_agent (14.853 s) : 14853000, 14853000
. : milestone, 14853000,
appsec (14.921 s) : 14921000, 14921000
. : milestone, 14921000,
iast (18.815 s) : 18815000, 18815000
. : milestone, 18815000,
iast_GLOBAL (18.126 s) : 18126000, 18126000
. : milestone, 18126000,
profiling (15.03 s) : 15030000, 15030000
. : milestone, 15030000,
tracing (14.993 s) : 14993000, 14993000
. : milestone, 14993000,
section candidate
no_agent (15.367 s) : 15367000, 15367000
. : milestone, 15367000,
appsec (15.074 s) : 15074000, 15074000
. : milestone, 15074000,
iast (18.446 s) : 18446000, 18446000
. : milestone, 18446000,
iast_GLOBAL (18.22 s) : 18220000, 18220000
. : milestone, 18220000,
profiling (15.884 s) : 15884000, 15884000
. : milestone, 15884000,
tracing (15.015 s) : 15015000, 15015000
. : milestone, 15015000,
|
# Conflicts: # dd-java-agent/build.gradle
For logs like ``` [main] WARN datadog.trace.bootstrap.AgentJarIndex - Detected duplicate content 'datadog.trace.civisibility.writer.ddintake.*' under 'trace', already seen in <root>. Ensure your content is under a distinct directory. ```
For logs like ``` Detected duplicate content under 'META-INF.AL2.0'. Ensure your content is under a distinct directory. Detected duplicate content under 'META-INF.LGPL2.1'. Ensure your content is under a distinct directory. ```
…nt files For logs like ``` [main] WARN datadog.trace.bootstrap.AgentJarIndex - Detected duplicate content under 'datadog.compiler.utils.*'. Ensure your content is under a distinct directory. [main] WARN datadog.trace.bootstrap.AgentJarIndex - Detected duplicate content under 'datadog.compiler.annotations.*'. Ensure your content is under a distinct ```
bric3
force-pushed
the
bdu/gen-index-tasks
branch
from
a15273d to
acea296
Compare
…etree.builtBy call
bric3
force-pushed
the
bdu/gen-index-tasks
branch
from
acea296 to
5dfca97
Compare
bric3
commented
Oct 13, 2025
| exclude(dependency('org.ow2.asm::')) | ||
|
|
||
| // javac plugin client | ||
| exclude(dependency('com.datadoghq:dd-javac-plugin-client')) |
Contributor
Author
There was a problem hiding this comment.
note: Fix repeated CompilerUtils and annotations being duplicated in every sub-agents
bric3
commented
Oct 13, 2025
Comment on lines
+27
to
+31
| // exclude 'jna' this since it's available in the instrumentation jar | ||
| dependencies { | ||
| exclude(dependency("net.java.dev.jna:jna")) | ||
| exclude(dependency("net.java.dev.jna:jna-platform")) | ||
| } |
Contributor
Author
There was a problem hiding this comment.
note: The previous exclusion forgot other files in META-INF
amarziali
reviewed
Oct 14, 2025
dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/AgentJarIndex.java
Outdated
Show resolved
Hide resolved
amarziali
reviewed
Oct 14, 2025
amarziali
approved these changes
Oct 14, 2025
Contributor
amarziali
left a comment
amarziali
left a comment
There was a problem hiding this comment.
I don't have a full overview of this part of the code. However the changes done looks good to me
Contributor
Author
|
@amarziali The issue with the previous code was that, the generation task was writing at the same location as other tasks, since multiple tasks could write at the same location, this break up-to-date checks, and as such the build cache for the impacted tasks. This PR properly segregate the output folder for each participating tasks (the jar extract and the index gen), this is is done by configuring the |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Refactor indexing tasks to avoid overlapping outputs.
Overlapping outputs prevent Gradle to perform correct up-to-date checks. As such it can results in either accidental dependencies, or force Gradle to re-execute the tasks.
Motivation
Speed-up the build.
Tasks with overlapping output (i.e. tasks that write to the same location, like the same parent folder), defeat Gradle build cache.
While doing so, it was discovered that the jar had duplicated classes in the various sub agents.
E.g. in 1.54.0
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any useful labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]