Migrate OtelContext wrapper to new Context API #8530

mcculls · 2025-03-09T22:16:21Z

Additional Notes

Tests are expected to fail while the scope manager has not yet been replaced.

That's why the final target of this stacked PR is the project/context feature branch.

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: APMAPI-855

pr-commenter · 2025-03-09T22:57:16Z

Benchmarks

Startup

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-03-10T15:21:54	2025-03-10T15:29:41
git_branch	master	mcculls/use-context-for-otel-shim
git_commit_date	1741604504	1741619390
git_commit_sha	`a464b9b`	`8305d62`
release_version	1.48.0-SNAPSHOT~a464b9bfb9	1.48.0-SNAPSHOT~8305d628cb
start_time	2025-03-10T15:21:40	2025-03-10T15:29:27

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1741620982	1741620982
ci_job_id	840331668	840331668
ci_pipeline_id	58285591	58285591
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-cq9dgfmp-project-304-concurrent-0-w0gj4rg4 6.8.0-1023-aws #25~22.04.1-Ubuntu SMP Tue Jan 28 12:51:22 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-cq9dgfmp-project-304-concurrent-0-w0gj4rg4 6.8.0-1023-aws #25~22.04.1-Ubuntu SMP Tue Jan 28 12:51:22 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 14 metrics, 16 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~8305d628cb, baseline=1.48.0-SNAPSHOT~a464b9bfb9
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.378 ms) : 1358, 1398
.   : milestone, 1378,
appsec (1.733 ms) : 1709, 1757
.   : milestone, 1733,
appsec_no_iast (1.769 ms) : 1745, 1793
.   : milestone, 1769,
code_origins (1.687 ms) : 1654, 1720
.   : milestone, 1687,
iast (1.52 ms) : 1495, 1544
.   : milestone, 1520,
profiling (1.533 ms) : 1510, 1556
.   : milestone, 1533,
tracing (1.486 ms) : 1460, 1512
.   : milestone, 1486,
section candidate
no_agent (1.368 ms) : 1348, 1388
.   : milestone, 1368,
appsec (1.738 ms) : 1713, 1762
.   : milestone, 1738,
appsec_no_iast (1.753 ms) : 1729, 1777
.   : milestone, 1753,
code_origins (1.688 ms) : 1656, 1721
.   : milestone, 1688,
iast (1.518 ms) : 1493, 1542
.   : milestone, 1518,
profiling (1.563 ms) : 1538, 1588
.   : milestone, 1563,
tracing (1.502 ms) : 1477, 1526
.   : milestone, 1502,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.378 ms [1.358 ms, 1.398 ms]	-
appsec	1.733 ms [1.709 ms, 1.757 ms]	354.968 µs (25.8%)
appsec_no_iast	1.769 ms [1.745 ms, 1.793 ms]	390.936 µs (28.4%)
code_origins	1.687 ms [1.654 ms, 1.72 ms]	309.038 µs (22.4%)
iast	1.52 ms [1.495 ms, 1.544 ms]	141.696 µs (10.3%)
profiling	1.533 ms [1.51 ms, 1.556 ms]	155.432 µs (11.3%)
tracing	1.486 ms [1.46 ms, 1.512 ms]	107.916 µs (7.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.368 ms [1.348 ms, 1.388 ms]	-
appsec	1.738 ms [1.713 ms, 1.762 ms]	369.428 µs (27.0%)
appsec_no_iast	1.753 ms [1.729 ms, 1.777 ms]	384.943 µs (28.1%)
code_origins	1.688 ms [1.656 ms, 1.721 ms]	320.207 µs (23.4%)
iast	1.518 ms [1.493 ms, 1.542 ms]	149.305 µs (10.9%)
profiling	1.563 ms [1.538 ms, 1.588 ms]	194.87 µs (14.2%)
tracing	1.502 ms [1.477 ms, 1.526 ms]	133.282 µs (9.7%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~8305d628cb, baseline=1.48.0-SNAPSHOT~a464b9bfb9
    dateFormat X
    axisFormat %s
section baseline
no_agent (386.367 µs) : 367, 406
.   : milestone, 386,
iast (519.741 µs) : 498, 541
.   : milestone, 520,
iast_FULL (738.583 µs) : 717, 761
.   : milestone, 739,
iast_GLOBAL (567.537 µs) : 545, 590
.   : milestone, 568,
iast_HARDCODED_SECRET_DISABLED (520.411 µs) : 499, 542
.   : milestone, 520,
iast_INACTIVE (471.656 µs) : 450, 493
.   : milestone, 472,
iast_TELEMETRY_OFF (512.497 µs) : 490, 535
.   : milestone, 512,
tracing (467.317 µs) : 446, 489
.   : milestone, 467,
section candidate
no_agent (387.373 µs) : 367, 407
.   : milestone, 387,
iast (522.817 µs) : 500, 546
.   : milestone, 523,
iast_FULL (735.848 µs) : 714, 758
.   : milestone, 736,
iast_GLOBAL (565.841 µs) : 543, 588
.   : milestone, 566,
iast_HARDCODED_SECRET_DISABLED (518.734 µs) : 497, 540
.   : milestone, 519,
iast_INACTIVE (469.796 µs) : 449, 491
.   : milestone, 470,
iast_TELEMETRY_OFF (501.849 µs) : 480, 523
.   : milestone, 502,
tracing (471.13 µs) : 450, 492
.   : milestone, 471,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	386.367 µs [366.769 µs, 405.965 µs]	-
iast	519.741 µs [498.152 µs, 541.33 µs]	133.374 µs (34.5%)
iast_FULL	738.583 µs [716.657 µs, 760.509 µs]	352.216 µs (91.2%)
iast_GLOBAL	567.537 µs [545.355 µs, 589.718 µs]	181.17 µs (46.9%)
iast_HARDCODED_SECRET_DISABLED	520.411 µs [498.808 µs, 542.015 µs]	134.044 µs (34.7%)
iast_INACTIVE	471.656 µs [449.926 µs, 493.386 µs]	85.289 µs (22.1%)
iast_TELEMETRY_OFF	512.497 µs [489.603 µs, 535.39 µs]	126.13 µs (32.6%)
tracing	467.317 µs [445.952 µs, 488.682 µs]	80.95 µs (21.0%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	387.373 µs [367.48 µs, 407.265 µs]	-
iast	522.817 µs [499.761 µs, 545.872 µs]	135.444 µs (35.0%)
iast_FULL	735.848 µs [713.986 µs, 757.711 µs]	348.476 µs (90.0%)
iast_GLOBAL	565.841 µs [543.254 µs, 588.428 µs]	178.468 µs (46.1%)
iast_HARDCODED_SECRET_DISABLED	518.734 µs [497.408 µs, 540.06 µs]	131.361 µs (33.9%)
iast_INACTIVE	469.796 µs [448.698 µs, 490.893 µs]	82.423 µs (21.3%)
iast_TELEMETRY_OFF	501.849 µs [480.25 µs, 523.448 µs]	114.476 µs (29.6%)
tracing	471.13 µs [450.158 µs, 492.102 µs]	83.757 µs (21.6%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	mcculls/use-context-for-otel-shim
git_commit_date	1741604504	1741619390
git_commit_sha	`a464b9b`	`8305d62`
release_version	1.48.0-SNAPSHOT~a464b9bfb9	1.48.0-SNAPSHOT~8305d628cb

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1741621454	1741621454
ci_job_id	840331673	840331673
ci_pipeline_id	58285591	58285591
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-5uqttet-project-304-concurrent-0-jp6o4qqo 6.8.0-1023-aws #25~22.04.1-Ubuntu SMP Tue Jan 28 12:51:22 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-5uqttet-project-304-concurrent-0-jp6o4qqo 6.8.0-1023-aws #25~22.04.1-Ubuntu SMP Tue Jan 28 12:51:22 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~8305d628cb, baseline=1.48.0-SNAPSHOT~a464b9bfb9
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.082 s) : 15082000, 15082000
.   : milestone, 15082000,
appsec (14.926 s) : 14926000, 14926000
.   : milestone, 14926000,
iast (18.98 s) : 18980000, 18980000
.   : milestone, 18980000,
iast_GLOBAL (17.936 s) : 17936000, 17936000
.   : milestone, 17936000,
profiling (15.665 s) : 15665000, 15665000
.   : milestone, 15665000,
tracing (14.852 s) : 14852000, 14852000
.   : milestone, 14852000,
section candidate
no_agent (15.364 s) : 15364000, 15364000
.   : milestone, 15364000,
appsec (15.177 s) : 15177000, 15177000
.   : milestone, 15177000,
iast (18.355 s) : 18355000, 18355000
.   : milestone, 18355000,
iast_GLOBAL (17.805 s) : 17805000, 17805000
.   : milestone, 17805000,
profiling (15.66 s) : 15660000, 15660000
.   : milestone, 15660000,
tracing (15.066 s) : 15066000, 15066000
.   : milestone, 15066000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.082 s [15.082 s, 15.082 s]	-
appsec	14.926 s [14.926 s, 14.926 s]	-156.0 ms (-1.0%)
iast	18.98 s [18.98 s, 18.98 s]	3.898 s (25.8%)
iast_GLOBAL	17.936 s [17.936 s, 17.936 s]	2.854 s (18.9%)
profiling	15.665 s [15.665 s, 15.665 s]	583.0 ms (3.9%)
tracing	14.852 s [14.852 s, 14.852 s]	-230.0 ms (-1.5%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.364 s [15.364 s, 15.364 s]	-
appsec	15.177 s [15.177 s, 15.177 s]	-187.0 ms (-1.2%)
iast	18.355 s [18.355 s, 18.355 s]	2.991 s (19.5%)
iast_GLOBAL	17.805 s [17.805 s, 17.805 s]	2.441 s (15.9%)
profiling	15.66 s [15.66 s, 15.66 s]	296.0 ms (1.9%)
tracing	15.066 s [15.066 s, 15.066 s]	-298.0 ms (-1.9%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~8305d628cb, baseline=1.48.0-SNAPSHOT~a464b9bfb9
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.47 ms) : 1459, 1482
.   : milestone, 1470,
appsec (2.325 ms) : 2281, 2369
.   : milestone, 2325,
iast (2.113 ms) : 2057, 2169
.   : milestone, 2113,
iast_GLOBAL (2.159 ms) : 2103, 2215
.   : milestone, 2159,
profiling (1.964 ms) : 1920, 2008
.   : milestone, 1964,
tracing (1.953 ms) : 1910, 1996
.   : milestone, 1953,
section candidate
no_agent (1.474 ms) : 1463, 1486
.   : milestone, 1474,
appsec (2.328 ms) : 2285, 2372
.   : milestone, 2328,
iast (2.105 ms) : 2050, 2159
.   : milestone, 2105,
iast_GLOBAL (2.154 ms) : 2099, 2210
.   : milestone, 2154,
profiling (1.96 ms) : 1917, 2003
.   : milestone, 1960,
tracing (1.949 ms) : 1907, 1992
.   : milestone, 1949,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.47 ms [1.459 ms, 1.482 ms]	-
appsec	2.325 ms [2.281 ms, 2.369 ms]	854.703 µs (58.1%)
iast	2.113 ms [2.057 ms, 2.169 ms]	642.904 µs (43.7%)
iast_GLOBAL	2.159 ms [2.103 ms, 2.215 ms]	688.925 µs (46.9%)
profiling	1.964 ms [1.92 ms, 2.008 ms]	493.912 µs (33.6%)
tracing	1.953 ms [1.91 ms, 1.996 ms]	482.63 µs (32.8%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.474 ms [1.463 ms, 1.486 ms]	-
appsec	2.328 ms [2.285 ms, 2.372 ms]	854.108 µs (57.9%)
iast	2.105 ms [2.05 ms, 2.159 ms]	630.568 µs (42.8%)
iast_GLOBAL	2.154 ms [2.099 ms, 2.21 ms]	680.319 µs (46.2%)
profiling	1.96 ms [1.917 ms, 2.003 ms]	485.798 µs (33.0%)
tracing	1.949 ms [1.907 ms, 1.992 ms]	475.048 µs (32.2%)

PerfectSlayer

I left few minor comments.

About the test changes, are they put in place because context tracking is missing and resources are leaking as the tests can run to the end? Because the cleanup seem to be already part of the test steps.
If so, should we really merge them as it will get confusing once the tests are working again properly?

...agent/agent-otel/otel-shim/src/main/java/datadog/opentelemetry/shim/context/OtelContext.java

...ent/agent-otel/otel-shim/src/main/java/datadog/opentelemetry/shim/trace/OtelSpanBuilder.java

mcculls · 2025-03-10T13:15:29Z

About the test changes, are they put in place because context tracking is missing and resources are leaking as the tests can run to the end? Because the cleanup seem to be already part of the test steps.

So the test changes here are adding missing cleanup of scopes/spans that's done for other tests in the class but was missed here. These tests will fail at the moment because we haven't finished the span <-> context coupling, so while OtelContext is now context-based it needs the scope manager to be fully replaced in order for the underlying context to reflect spans being activated, etc. using our internal API.

However there's a difference between a test failing (which helps us focus on what remains to be fixed) and it causing unrelated failures in other tests (which would pass without the bad scope state leaking from the failing tests.)

So these are really test fixes, to ensure failing tests don't cause unrelated tests to fail.

( in other words, we'd still want to keep them because they make sure test failures don't leak to the other tests )

mcculls added tag: no release notes Changes to exclude from release notes type: refactoring inst: opentelemetry OpenTelemetry instrumentation labels Mar 9, 2025

mcculls requested a review from PerfectSlayer March 9, 2025 22:26

PerfectSlayer reviewed Mar 10, 2025

View reviewed changes

mcculls force-pushed the mcculls/remove-scopestateaware-api branch from 5e870d7 to 7724a95 Compare March 10, 2025 13:05

Base automatically changed from mcculls/remove-scopestateaware-api to project/context March 10, 2025 13:06

Migrate OtelContext wrapper to new Context API

591efc8

mcculls force-pushed the mcculls/use-context-for-otel-shim branch from 5c793f8 to 591efc8 Compare March 10, 2025 13:06

review feedback

8305d62

mcculls marked this pull request as ready for review March 10, 2025 15:20

mcculls requested review from a team as code owners March 10, 2025 15:20

mcculls requested review from dougqh and removed request for dougqh March 10, 2025 15:21

mcculls merged commit 819daaa into project/context Mar 10, 2025
189 of 209 checks passed

mcculls deleted the mcculls/use-context-for-otel-shim branch March 10, 2025 16:06

mcculls added a commit that referenced this pull request Mar 15, 2025

Migrate OtelContext wrapper to new Context API (#8530)

3236f5a

mcculls added a commit that referenced this pull request Mar 17, 2025

Migrate OtelContext wrapper to new Context API (#8530)

d0b06b4

mcculls added a commit that referenced this pull request Mar 18, 2025

Migrate OtelContext wrapper to new Context API (#8530)

8a505b2

mcculls added a commit that referenced this pull request Mar 18, 2025

Migrate OtelContext wrapper to new Context API (#8530)

725ef5e

mcculls added a commit that referenced this pull request Mar 18, 2025

Migrate OtelContext wrapper to new Context API (#8530)

acc32a5

mcculls added a commit that referenced this pull request Mar 19, 2025

Migrate OtelContext wrapper to new Context API (#8530)

a5e925f

mcculls added a commit that referenced this pull request Mar 21, 2025

Migrate OtelContext wrapper to new Context API (#8530)

3ad0629

mcculls added a commit that referenced this pull request Mar 26, 2025

Migrate OtelContext wrapper to new Context API (#8530)

a1c57c4

mcculls mentioned this pull request Mar 31, 2025

Migrate OtelContext wrapper to new internal Context API #8645

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Migrate OtelContext wrapper to new Context API #8530

Migrate OtelContext wrapper to new Context API #8530

Uh oh!

mcculls commented Mar 9, 2025 •

edited by jira bot

Loading

Uh oh!

pr-commenter bot commented Mar 9, 2025 •

edited

Loading

Uh oh!

PerfectSlayer left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mcculls commented Mar 10, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Migrate OtelContext wrapper to new Context API #8530

Migrate OtelContext wrapper to new Context API #8530

Uh oh!

Conversation

mcculls commented Mar 9, 2025 • edited by jira bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented Mar 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

PerfectSlayer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mcculls commented Mar 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mcculls commented Mar 9, 2025 •

edited by jira bot

Loading

pr-commenter bot commented Mar 9, 2025 •

edited

Loading

mcculls commented Mar 10, 2025 •

edited

Loading