Skip to content

Add single parameter dd.data.jobs.enabled to enable DJM #6972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
May 7, 2024
Merged

Add single parameter dd.data.jobs.enabled to enable DJM #6972

merged 4 commits into from
May 7, 2024

Conversation

@paul-laffon-dd
Copy link
Contributor

@paul-laffon-dd paul-laffon-dd commented Apr 30, 2024 •
edited
Loading

What Does This Do

Add single parameter dd.data.jobs.enabled=true to enable data jobs monitoring.

This parameter will change the default to have

  • dd.trace.experimental.long-running.enabled=true
  • dd.integration.spark.enabled=true
  • dd.integration.spark-executor.enabled=true

Only the default are changed, those parameters can still be changed by explicitly providing them

Motivation

Simpler way to enabled DJM

Additional Notes

@paul-laffon-dd paul-laffon-dd force-pushed the paul.laffon/data-jobs-enabled branch from 6a90b7e to 6afc1c7 Compare April 30, 2024 09:38
@pr-commenter
Copy link

pr-commenter bot commented Apr 30, 2024 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master paul.laffon/data-jobs-enabled
git_commit_date 1715068887 1715069313
git_commit_sha 6978c8e d703e90
release_version 1.34.0-SNAPSHOT~6978c8eed3 1.34.0-SNAPSHOT~d703e9037c
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1715072043 1715072043
ci_job_id 505811816 505811816
ci_pipeline_id 33741135 33741135
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 48 metrics, 15 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.34.0-SNAPSHOT~d703e9037c, baseline=1.34.0-SNAPSHOT~6978c8eed3

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.08 s) : 0, 1079819
Total [baseline] (8.546 s) : 0, 8546106
Agent [candidate] (1.08 s) : 0, 1079885
Total [candidate] (8.64 s) : 0, 8639778
section iast
Agent [baseline] (1.211 s) : 0, 1211146
Total [baseline] (9.003 s) : 0, 9003063
Agent [candidate] (1.206 s) : 0, 1206257
Total [candidate] (9.049 s) : 0, 9049499
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.203 s) : 0, 1202695
Total [baseline] (8.996 s) : 0, 8995824
Agent [candidate] (1.213 s) : 0, 1213072
Total [candidate] (9.061 s) : 0, 9060884
section iast_TELEMETRY_OFF
Agent [baseline] (1.199 s) : 0, 1199052
Total [baseline] (9.003 s) : 0, 9003016
Agent [candidate] (1.2 s) : 0, 1199982
Total [candidate] (9.007 s) : 0, 9006583
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.08 s -
Agent iast 1.211 s 131.327 ms (12.2%)
Agent iast_HARDCODED_SECRET_DISABLED 1.203 s 122.876 ms (11.4%)
Agent iast_TELEMETRY_OFF 1.199 s 119.233 ms (11.0%)
Total tracing 8.546 s -
Total iast 9.003 s 456.957 ms (5.3%)
Total iast_HARDCODED_SECRET_DISABLED 8.996 s 449.718 ms (5.3%)
Total iast_TELEMETRY_OFF 9.003 s 456.909 ms (5.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.08 s -
Agent iast 1.206 s 126.372 ms (11.7%)
Agent iast_HARDCODED_SECRET_DISABLED 1.213 s 133.187 ms (12.3%)
Agent iast_TELEMETRY_OFF 1.2 s 120.097 ms (11.1%)
Total tracing 8.64 s -
Total iast 9.049 s 409.721 ms (4.7%)
Total iast_HARDCODED_SECRET_DISABLED 9.061 s 421.106 ms (4.9%)
Total iast_TELEMETRY_OFF 9.007 s 366.805 ms (4.2%) 
gantt
    title insecure-bank - break down per module: candidate=1.34.0-SNAPSHOT~d703e9037c, baseline=1.34.0-SNAPSHOT~6978c8eed3

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (675.894 ms) : 0, 675894
BytebuddyAgent [candidate] (675.689 ms) : 0, 675689
GlobalTracer [baseline] (311.66 ms) : 0, 311660
GlobalTracer [candidate] (311.67 ms) : 0, 311670
AppSec [baseline] (49.556 ms) : 0, 49556
AppSec [candidate] (49.728 ms) : 0, 49728
Remote Config [baseline] (662.151 µs) : 0, 662
Remote Config [candidate] (671.809 µs) : 0, 672
Telemetry [baseline] (7.592 ms) : 0, 7592
Telemetry [candidate] (7.677 ms) : 0, 7677
section iast
BytebuddyAgent [baseline] (802.603 ms) : 0, 802603
BytebuddyAgent [candidate] (797.734 ms) : 0, 797734
GlobalTracer [baseline] (292.143 ms) : 0, 292143
GlobalTracer [candidate] (291.001 ms) : 0, 291001
AppSec [baseline] (51.055 ms) : 0, 51055
AppSec [candidate] (49.145 ms) : 0, 49145
IAST [baseline] (23.477 ms) : 0, 23477
IAST [candidate] (26.02 ms) : 0, 26020
Remote Config [baseline] (577.177 µs) : 0, 577
Remote Config [candidate] (581.019 µs) : 0, 581
Telemetry [baseline] (6.64 ms) : 0, 6640
Telemetry [candidate] (7.313 ms) : 0, 7313
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (795.282 ms) : 0, 795282
BytebuddyAgent [candidate] (802.032 ms) : 0, 802032
GlobalTracer [baseline] (290.317 ms) : 0, 290317
GlobalTracer [candidate] (292.63 ms) : 0, 292630
AppSec [baseline] (51.31 ms) : 0, 51310
AppSec [candidate] (53.423 ms) : 0, 53423
IAST [baseline] (23.402 ms) : 0, 23402
IAST [candidate] (23.08 ms) : 0, 23080
Remote Config [baseline] (587.391 µs) : 0, 587
Remote Config [candidate] (582.843 µs) : 0, 583
Telemetry [baseline] (7.361 ms) : 0, 7361
Telemetry [candidate] (6.74 ms) : 0, 6740
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (792.717 ms) : 0, 792717
BytebuddyAgent [candidate] (793.148 ms) : 0, 793148
GlobalTracer [baseline] (290.283 ms) : 0, 290283
GlobalTracer [candidate] (290.36 ms) : 0, 290360
AppSec [baseline] (51.719 ms) : 0, 51719
AppSec [candidate] (51.242 ms) : 0, 51242
IAST [baseline] (22.031 ms) : 0, 22031
IAST [candidate] (23.66 ms) : 0, 23660
Remote Config [baseline] (586.286 µs) : 0, 586
Remote Config [candidate] (581.226 µs) : 0, 581
Telemetry [baseline] (7.33 ms) : 0, 7330
Telemetry [candidate] (6.58 ms) : 0, 6580
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.34.0-SNAPSHOT~d703e9037c, baseline=1.34.0-SNAPSHOT~6978c8eed3

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.088 s) : 0, 1087830
Total [baseline] (10.358 s) : 0, 10357651
Agent [candidate] (1.085 s) : 0, 1084546
Total [candidate] (10.413 s) : 0, 10413422
section appsec
Agent [baseline] (1.204 s) : 0, 1204278
Total [baseline] (10.521 s) : 0, 10520632
Agent [candidate] (1.195 s) : 0, 1194797
Total [candidate] (10.459 s) : 0, 10458508
section iast
Agent [baseline] (1.21 s) : 0, 1210152
Total [baseline] (10.778 s) : 0, 10777640
Agent [candidate] (1.203 s) : 0, 1202584
Total [candidate] (10.811 s) : 0, 10811433
section profiling
Agent [baseline] (1.274 s) : 0, 1273863
Total [baseline] (10.601 s) : 0, 10600758
Agent [candidate] (1.29 s) : 0, 1290412
Total [candidate] (10.604 s) : 0, 10603553
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.088 s -
Agent appsec 1.204 s 116.448 ms (10.7%)
Agent iast 1.21 s 122.322 ms (11.2%)
Agent profiling 1.274 s 186.033 ms (17.1%)
Total tracing 10.358 s -
Total appsec 10.521 s 162.982 ms (1.6%)
Total iast 10.778 s 419.99 ms (4.1%)
Total profiling 10.601 s 243.107 ms (2.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.085 s -
Agent appsec 1.195 s 110.251 ms (10.2%)
Agent iast 1.203 s 118.038 ms (10.9%)
Agent profiling 1.29 s 205.865 ms (19.0%)
Total tracing 10.413 s -
Total appsec 10.459 s 45.086 ms (0.4%)
Total iast 10.811 s 398.011 ms (3.8%)
Total profiling 10.604 s 190.13 ms (1.8%) 
gantt
    title petclinic - break down per module: candidate=1.34.0-SNAPSHOT~d703e9037c, baseline=1.34.0-SNAPSHOT~6978c8eed3

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (681.166 ms) : 0, 681166
BytebuddyAgent [candidate] (679.066 ms) : 0, 679066
GlobalTracer [baseline] (313.577 ms) : 0, 313577
GlobalTracer [candidate] (312.807 ms) : 0, 312807
AppSec [baseline] (50.026 ms) : 0, 50026
AppSec [candidate] (49.687 ms) : 0, 49687
Remote Config [baseline] (677.889 µs) : 0, 678
Remote Config [candidate] (672.761 µs) : 0, 673
Telemetry [baseline] (7.709 ms) : 0, 7709
Telemetry [candidate] (7.667 ms) : 0, 7667
section appsec
BytebuddyAgent [baseline] (703.718 ms) : 0, 703718
BytebuddyAgent [candidate] (697.95 ms) : 0, 697950
GlobalTracer [baseline] (296.28 ms) : 0, 296280
GlobalTracer [candidate] (293.965 ms) : 0, 293965
AppSec [baseline] (150.063 ms) : 0, 150063
AppSec [candidate] (149.297 ms) : 0, 149297
Remote Config [baseline] (628.01 µs) : 0, 628
Remote Config [candidate] (627.277 µs) : 0, 627
Telemetry [baseline] (9.115 ms) : 0, 9115
Telemetry [candidate] (8.799 ms) : 0, 8799
IAST [baseline] (19.439 ms) : 0, 19439
IAST [candidate] (19.313 ms) : 0, 19313
section iast
BytebuddyAgent [baseline] (801.013 ms) : 0, 801013
BytebuddyAgent [candidate] (795.371 ms) : 0, 795371
GlobalTracer [baseline] (291.918 ms) : 0, 291918
GlobalTracer [candidate] (290.314 ms) : 0, 290314
AppSec [baseline] (49.691 ms) : 0, 49691
AppSec [candidate] (51.059 ms) : 0, 51059
Remote Config [baseline] (566.683 µs) : 0, 567
Remote Config [candidate] (568.437 µs) : 0, 568
Telemetry [baseline] (6.586 ms) : 0, 6586
Telemetry [candidate] (7.408 ms) : 0, 7408
IAST [baseline] (25.713 ms) : 0, 25713
IAST [candidate] (23.454 ms) : 0, 23454
section profiling
BytebuddyAgent [baseline] (681.006 ms) : 0, 681006
BytebuddyAgent [candidate] (689.088 ms) : 0, 689088
GlobalTracer [baseline] (381.385 ms) : 0, 381385
GlobalTracer [candidate] (387.037 ms) : 0, 387037
AppSec [baseline] (50.161 ms) : 0, 50161
AppSec [candidate] (51.16 ms) : 0, 51160
Remote Config [baseline] (719.681 µs) : 0, 720
Remote Config [candidate] (729.074 µs) : 0, 729
Telemetry [baseline] (7.456 ms) : 0, 7456
Telemetry [candidate] (7.569 ms) : 0, 7569
ProfilingAgent [baseline] (96.122 ms) : 0, 96122
ProfilingAgent [candidate] (97.399 ms) : 0, 97399
Profiling [baseline] (96.147 ms) : 0, 96147
Profiling [candidate] (97.424 ms) : 0, 97424
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-05-07T08:26:29 2024-05-07T08:33:18
git_branch master paul.laffon/data-jobs-enabled
git_commit_date 1715068887 1715069313
git_commit_sha 6978c8e d703e90
release_version 1.34.0-SNAPSHOT~6978c8eed3 1.34.0-SNAPSHOT~d703e9037c
start_time 2024-05-07T08:26:15 2024-05-07T08:33:05
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1715071144 1715071144
ci_job_id 505811817 505811817
ci_pipeline_id 33741135 33741135
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.34.0-SNAPSHOT~d703e9037c, baseline=1.34.0-SNAPSHOT~6978c8eed3
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.337 ms) : 1319, 1356
.   : milestone, 1337,
appsec (1.731 ms) : 1708, 1755
.   : milestone, 1731,
appsec_no_iast (1.706 ms) : 1681, 1730
.   : milestone, 1706,
iast (1.488 ms) : 1466, 1511
.   : milestone, 1488,
profiling (1.546 ms) : 1520, 1571
.   : milestone, 1546,
tracing (1.479 ms) : 1455, 1502
.   : milestone, 1479,
section candidate
no_agent (1.357 ms) : 1337, 1378
.   : milestone, 1357,
appsec (1.711 ms) : 1687, 1735
.   : milestone, 1711,
appsec_no_iast (1.708 ms) : 1683, 1733
.   : milestone, 1708,
iast (1.486 ms) : 1463, 1508
.   : milestone, 1486,
profiling (1.55 ms) : 1524, 1575
.   : milestone, 1550,
tracing (1.488 ms) : 1463, 1513
.   : milestone, 1488,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.337 ms [1.319 ms, 1.356 ms] -
appsec 1.731 ms [1.708 ms, 1.755 ms] 394.295 µs (29.5%)
appsec_no_iast 1.706 ms [1.681 ms, 1.73 ms] 368.352 µs (27.5%)
iast 1.488 ms [1.466 ms, 1.511 ms] 151.245 µs (11.3%)
profiling 1.546 ms [1.52 ms, 1.571 ms] 208.495 µs (15.6%)
tracing 1.479 ms [1.455 ms, 1.502 ms] 141.746 µs (10.6%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.357 ms [1.337 ms, 1.378 ms] -
appsec 1.711 ms [1.687 ms, 1.735 ms] 353.508 µs (26.0%)
appsec_no_iast 1.708 ms [1.683 ms, 1.733 ms] 350.917 µs (25.9%)
iast 1.486 ms [1.463 ms, 1.508 ms] 128.222 µs (9.4%)
profiling 1.55 ms [1.524 ms, 1.575 ms] 192.243 µs (14.2%)
tracing 1.488 ms [1.463 ms, 1.513 ms] 130.837 µs (9.6%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.34.0-SNAPSHOT~d703e9037c, baseline=1.34.0-SNAPSHOT~6978c8eed3
    dateFormat X
    axisFormat %s
section baseline
no_agent (374.436 µs) : 355, 394
.   : milestone, 374,
iast (488.471 µs) : 468, 509
.   : milestone, 488,
iast_FULL (546.087 µs) : 525, 567
.   : milestone, 546,
iast_GLOBAL (496.215 µs) : 476, 517
.   : milestone, 496,
iast_HARDCODED_SECRET_DISABLED (477.138 µs) : 456, 498
.   : milestone, 477,
iast_INACTIVE (441.328 µs) : 421, 462
.   : milestone, 441,
iast_TELEMETRY_OFF (473.948 µs) : 453, 495
.   : milestone, 474,
tracing (446.18 µs) : 424, 469
.   : milestone, 446,
section candidate
no_agent (373.314 µs) : 354, 393
.   : milestone, 373,
iast (476.491 µs) : 456, 497
.   : milestone, 476,
iast_FULL (541.995 µs) : 521, 563
.   : milestone, 542,
iast_GLOBAL (490.02 µs) : 469, 511
.   : milestone, 490,
iast_HARDCODED_SECRET_DISABLED (472.925 µs) : 452, 493
.   : milestone, 473,
iast_INACTIVE (449.26 µs) : 428, 470
.   : milestone, 449,
iast_TELEMETRY_OFF (469.678 µs) : 448, 491
.   : milestone, 470,
tracing (442.923 µs) : 423, 463
.   : milestone, 443,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 374.436 µs [355.027 µs, 393.845 µs] -
iast 488.471 µs [467.784 µs, 509.157 µs] 114.035 µs (30.5%)
iast_FULL 546.087 µs [525.156 µs, 567.018 µs] 171.651 µs (45.8%)
iast_GLOBAL 496.215 µs [475.525 µs, 516.905 µs] 121.779 µs (32.5%)
iast_HARDCODED_SECRET_DISABLED 477.138 µs [456.119 µs, 498.157 µs] 102.702 µs (27.4%)
iast_INACTIVE 441.328 µs [421.029 µs, 461.628 µs] 66.892 µs (17.9%)
iast_TELEMETRY_OFF 473.948 µs [452.836 µs, 495.06 µs] 99.512 µs (26.6%)
tracing 446.18 µs [423.58 µs, 468.78 µs] 71.744 µs (19.2%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 373.314 µs [353.858 µs, 392.77 µs] -
iast 476.491 µs [455.599 µs, 497.383 µs] 103.177 µs (27.6%)
iast_FULL 541.995 µs [521.454 µs, 562.536 µs] 168.681 µs (45.2%)
iast_GLOBAL 490.02 µs [469.39 µs, 510.649 µs] 116.706 µs (31.3%)
iast_HARDCODED_SECRET_DISABLED 472.925 µs [452.391 µs, 493.459 µs] 99.611 µs (26.7%)
iast_INACTIVE 449.26 µs [428.112 µs, 470.408 µs] 75.946 µs (20.3%)
iast_TELEMETRY_OFF 469.678 µs [448.317 µs, 491.039 µs] 96.364 µs (25.8%)
tracing 442.923 µs [422.728 µs, 463.118 µs] 69.609 µs (18.6%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master paul.laffon/data-jobs-enabled
git_commit_date 1715068887 1715069313
git_commit_sha 6978c8e d703e90
release_version 1.34.0-SNAPSHOT~6978c8eed3 1.34.0-SNAPSHOT~d703e9037c
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1715071662 1715071662
ci_job_id 505811818 505811818
ci_pipeline_id 33741135 33741135
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.34.0-SNAPSHOT~d703e9037c, baseline=1.34.0-SNAPSHOT~6978c8eed3
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.46 ms) : 1448, 1471
.   : milestone, 1460,
appsec (2.216 ms) : 2182, 2251
.   : milestone, 2216,
iast (1.881 ms) : 1846, 1915
.   : milestone, 1881,
iast_GLOBAL (1.927 ms) : 1891, 1962
.   : milestone, 1927,
profiling (2.294 ms) : 2125, 2463
.   : milestone, 2294,
tracing (1.827 ms) : 1796, 1859
.   : milestone, 1827,
section candidate
no_agent (1.464 ms) : 1453, 1475
.   : milestone, 1464,
appsec (2.209 ms) : 2175, 2243
.   : milestone, 2209,
iast (1.872 ms) : 1838, 1907
.   : milestone, 1872,
iast_GLOBAL (1.916 ms) : 1881, 1951
.   : milestone, 1916,
profiling (1.843 ms) : 1810, 1877
.   : milestone, 1843,
tracing (1.835 ms) : 1802, 1867
.   : milestone, 1835,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.46 ms [1.448 ms, 1.471 ms] -
appsec 2.216 ms [2.182 ms, 2.251 ms] 756.852 µs (51.9%)
iast 1.881 ms [1.846 ms, 1.915 ms] 421.217 µs (28.9%)
iast_GLOBAL 1.927 ms [1.891 ms, 1.962 ms] 467.118 µs (32.0%)
profiling 2.294 ms [2.125 ms, 2.463 ms] 834.487 µs (57.2%)
tracing 1.827 ms [1.796 ms, 1.859 ms] 367.751 µs (25.2%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.464 ms [1.453 ms, 1.475 ms] -
appsec 2.209 ms [2.175 ms, 2.243 ms] 745.018 µs (50.9%)
iast 1.872 ms [1.838 ms, 1.907 ms] 408.407 µs (27.9%)
iast_GLOBAL 1.916 ms [1.881 ms, 1.951 ms] 451.516 µs (30.8%)
profiling 1.843 ms [1.81 ms, 1.877 ms] 379.293 µs (25.9%)
tracing 1.835 ms [1.802 ms, 1.867 ms] 370.561 µs (25.3%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.34.0-SNAPSHOT~d703e9037c, baseline=1.34.0-SNAPSHOT~6978c8eed3
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.921 s) : 14921000, 14921000
.   : milestone, 14921000,
appsec (15.255 s) : 15255000, 15255000
.   : milestone, 15255000,
iast (18.957 s) : 18957000, 18957000
.   : milestone, 18957000,
iast_GLOBAL (17.959 s) : 17959000, 17959000
.   : milestone, 17959000,
profiling (15.484 s) : 15484000, 15484000
.   : milestone, 15484000,
tracing (15.181 s) : 15181000, 15181000
.   : milestone, 15181000,
section candidate
no_agent (14.932 s) : 14932000, 14932000
.   : milestone, 14932000,
appsec (15.147 s) : 15147000, 15147000
.   : milestone, 15147000,
iast (18.803 s) : 18803000, 18803000
.   : milestone, 18803000,
iast_GLOBAL (17.791 s) : 17791000, 17791000
.   : milestone, 17791000,
profiling (15.027 s) : 15027000, 15027000
.   : milestone, 15027000,
tracing (15.075 s) : 15075000, 15075000
.   : milestone, 15075000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.921 s [14.921 s, 14.921 s] -
appsec 15.255 s [15.255 s, 15.255 s] 334.0 ms (2.2%)
iast 18.957 s [18.957 s, 18.957 s] 4.036 s (27.0%)
iast_GLOBAL 17.959 s [17.959 s, 17.959 s] 3.038 s (20.4%)
profiling 15.484 s [15.484 s, 15.484 s] 563.0 ms (3.8%)
tracing 15.181 s [15.181 s, 15.181 s] 260.0 ms (1.7%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.932 s [14.932 s, 14.932 s] -
appsec 15.147 s [15.147 s, 15.147 s] 215.0 ms (1.4%)
iast 18.803 s [18.803 s, 18.803 s] 3.871 s (25.9%)
iast_GLOBAL 17.791 s [17.791 s, 17.791 s] 2.859 s (19.1%)
profiling 15.027 s [15.027 s, 15.027 s] 95.0 ms (0.6%)
tracing 15.075 s [15.075 s, 15.075 s] 143.0 ms (1.0%)

@paul-laffon-dd paul-laffon-dd force-pushed the paul.laffon/data-jobs-enabled branch from 6afc1c7 to a767b61 Compare May 2, 2024 08:37
@paul-laffon-dd paul-laffon-dd force-pushed the paul.laffon/data-jobs-enabled branch from a767b61 to 0af3ab0 Compare May 2, 2024 08:56
@paul-laffon-dd paul-laffon-dd marked this pull request as ready for review May 2, 2024 11:37
@paul-laffon-dd paul-laffon-dd requested a review from a team as a code owner May 2, 2024 11:37
@paul-laffon-dd paul-laffon-dd requested review from am312 and dougqh May 2, 2024 11:37
gabe-ochoa
gabe-ochoa reviewed May 2, 2024
View reviewed changes
dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/Agent.java
Comment on lines +211 to +214
setSystemPropertyDefault(
propertyNameToSystemPropertyName("integration.spark.enabled"), "true");
setSystemPropertyDefault(
propertyNameToSystemPropertyName("integration.spark-executor.enabled"), "true");
Copy link

@gabe-ochoa gabe-ochoa May 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once the apm intake work is done for #6967, can we just turn on all integrations?

Copy link
Contributor

@kr-igor kr-igor May 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to disable sampling as well? It may be a big issue for some hight throughput systems with millions of operations per second.

Copy link
Contributor Author

@paul-laffon-dd paul-laffon-dd May 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed it so that dd.data.jobs.enabled=true does not change the default for all integrations (enabled by default), only enables spark/long running spans

@gabe-ochoa gabe-ochoa mentioned this pull request May 2, 2024
Merged
kr-igor
kr-igor requested changes May 2, 2024
View reviewed changes
Copy link
Contributor

@kr-igor kr-igor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this change may cause some issues for customers who tries to use DSM in combination with DJM, as instrumentations required by DSM will be forcibly disabled. Could you also add "force enable" for DSM instrumentations if DSM_ENABLED = true? You can find these instrumentations by searching setDatastreamsCheckpoints.

dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/Agent.java Outdated
if (dataJobsEnabled) {
log.info("Data Jobs Monitoring enabled, changing default integrations enabled");
setSystemPropertyDefault(
propertyNameToSystemPropertyName(TraceInstrumentationConfig.INTEGRATIONS_ENABLED),
Copy link
Contributor

@kr-igor kr-igor Apr 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should not override user-defined preferences. I.e. if the user explicitly set integrations_enabled=true is should not be reset. However, if the user didn't set the option, the default option should be overwritten.

Copy link
Contributor Author

@paul-laffon-dd paul-laffon-dd May 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The setSystemPropertyDefault is only changing the default, if the user explicitly set a parameter it will take precedence over the default

@paul-laffon-dd paul-laffon-dd force-pushed the paul.laffon/data-jobs-enabled branch from 30017d2 to 57d83ad Compare May 3, 2024 10:30
@paul-laffon-dd paul-laffon-dd merged commit 2267ce3 into master May 7, 2024
@paul-laffon-dd paul-laffon-dd deleted the paul.laffon/data-jobs-enabled branch May 7, 2024 10:07
@github-actions github-actions bot added this to the 1.34.0 milestone May 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dougqh dougqh dougqh approved these changes

@kr-igor kr-igor kr-igor approved these changes

@yiliangzhou yiliangzhou Awaiting requested review from yiliangzhou

@am312 am312 Awaiting requested review from am312 am312 is a code owner automatically assigned from DataDog/apm-java

@gabe-ochoa gabe-ochoa Awaiting requested review from gabe-ochoa

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

1.34.0

Development

Successfully merging this pull request may close these issues.

5 participants

@paul-laffon-dd @dougqh @gabe-ochoa @kr-igor