[Tracing] Add experimental support for exporting traces as OTLP (.NET 6+)#8211
[Tracing] Add experimental support for exporting traces as OTLP (.NET 6+)#8211zacharycmontoya wants to merge 14 commits intomasterfrom
Conversation
54892c0 to
771b40f
Compare
BenchmarksBenchmark execution time: 2026-02-20 22:17:56 Comparing candidate commit fe6d013 in PR branch Found 13 performance improvements and 12 performance regressions! Performance is the same for 152 metrics, 15 unstable metrics. scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net472
scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net6.0
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net472
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net6.0
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody netcoreapp3.1
scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs netcoreapp3.1
scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net6.0
scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net6.0
scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net6.0
scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1
scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice netcoreapp3.1
scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net6.0
scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch netcoreapp3.1
scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net6.0
scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog net472
scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog netcoreapp3.1
scenario:Benchmarks.Trace.RedisBenchmark.SendReceive net472
scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope net6.0
scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan netcoreapp3.1
scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes net6.0
scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes netcoreapp3.1
|
…m actually sending traces
…all ISpanBufferSerializer/SpanBufferMessagePackSerializer class to enable the future OTLP serialization
…e SpanBuffer serialization.
…Buffer. This does not yet handle issuing the outbound HTTP requests through the Api classes, nor does it handle configuration to enable the feature"
…udes: - Making the explicit histogram bounds configurable - Make property setters available internally so we can construct the final snapshots from our DD Sketches (rather than building up the snapshots over time and reading them at export time)
…ay we must distinguish the histogram timeseries when submitting trace stats using OTLP metrics.
…g APM trace stats) configurations to dd-trace-dotnet and read them in ExporterSettings
…submission, coming in a following commit
…and traces via OTLP traces, and plug them in to TracerManagerFactory.GetAgentWriter so we can use the new feature end-to-end
…itsOtlpTraces. Currently only http/json for Traces and http/protobuf for Metrics is tested.
771b40f to
e729239
Compare
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (8211) and master.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Metric | Master (Mean ± 95% CI) | Current (Mean ± 95% CI) | Change | Status |
|---|---|---|---|---|
| .NET 6 - CallTarget+Inlining+NGEN | ||||
| runtime.dotnet.mem.committed | 57.52 ± (57.36 - 57.69) MB | 63.02 ± (62.97 - 63.07) MB | +9.6% | ❌⬆️ |
Full Metrics Comparison
FakeDbCommand
| Metric | Master (Mean ± 95% CI) | Current (Mean ± 95% CI) | Change | Status |
|---|---|---|---|---|
| .NET Framework 4.8 - Baseline | ||||
| duration | 76.29 ± (76.25 - 76.64) ms | 74.96 ± (74.84 - 75.11) ms | -1.7% | ✅ |
| .NET Framework 4.8 - Bailout | ||||
| duration | 80.28 ± (80.11 - 80.50) ms | 79.89 ± (79.74 - 80.13) ms | -0.5% | ✅ |
| .NET Framework 4.8 - CallTarget+Inlining+NGEN | ||||
| duration | 1096.91 ± (1095.68 - 1102.21) ms | 1092.37 ± (1093.61 - 1101.21) ms | -0.4% | ✅ |
| .NET Core 3.1 - Baseline | ||||
| process.internal_duration_ms | 23.27 ± (23.22 - 23.32) ms | 22.93 ± (22.89 - 22.98) ms | -1.5% | ✅ |
| process.time_to_main_ms | 88.90 ± (88.67 - 89.12) ms | 86.80 ± (86.60 - 87.00) ms | -2.4% | ✅ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.93 ± (10.93 - 10.94) MB | 10.91 ± (10.91 - 10.92) MB | -0.2% | ✅ |
| runtime.dotnet.threads.count | 12 ± (12 - 12) | 12 ± (12 - 12) | +0.0% | ✅ |
| .NET Core 3.1 - Bailout | ||||
| process.internal_duration_ms | 23.08 ± (23.02 - 23.14) ms | 22.78 ± (22.73 - 22.84) ms | -1.3% | ✅ |
| process.time_to_main_ms | 90.22 ± (90.04 - 90.40) ms | 87.99 ± (87.79 - 88.19) ms | -2.5% | ✅ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.95 ± (10.95 - 10.96) MB | 10.96 ± (10.95 - 10.96) MB | +0.1% | ✅⬆️ |
| runtime.dotnet.threads.count | 13 ± (13 - 13) | 13 ± (13 - 13) | +0.0% | ✅ |
| .NET Core 3.1 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 233.04 ± (229.37 - 236.70) ms | 247.88 ± (243.92 - 251.84) ms | +6.4% | ✅⬆️ |
| process.time_to_main_ms | 504.89 ± (504.10 - 505.68) ms | 496.53 ± (495.84 - 497.22) ms | -1.7% | ✅ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 47.60 ± (47.58 - 47.62) MB | 47.73 ± (47.71 - 47.75) MB | +0.3% | ✅⬆️ |
| runtime.dotnet.threads.count | 28 ± (28 - 28) | 28 ± (28 - 28) | +0.0% | ✅⬆️ |
| .NET 6 - Baseline | ||||
| process.internal_duration_ms | 22.01 ± (21.97 - 22.06) ms | 21.53 ± (21.49 - 21.58) ms | -2.2% | ✅ |
| process.time_to_main_ms | 78.03 ± (77.85 - 78.20) ms | 75.36 ± (75.18 - 75.54) ms | -3.4% | ✅ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.62 ± (10.62 - 10.62) MB | 10.63 ± (10.63 - 10.63) MB | +0.1% | ✅⬆️ |
| runtime.dotnet.threads.count | 10 ± (10 - 10) | 10 ± (10 - 10) | +0.0% | ✅ |
| .NET 6 - Bailout | ||||
| process.internal_duration_ms | 21.77 ± (21.71 - 21.82) ms | 21.49 ± (21.44 - 21.55) ms | -1.3% | ✅ |
| process.time_to_main_ms | 78.75 ± (78.56 - 78.95) ms | 76.50 ± (76.30 - 76.70) ms | -2.9% | ✅ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.67 ± (10.66 - 10.67) MB | 10.72 ± (10.71 - 10.72) MB | +0.5% | ✅⬆️ |
| runtime.dotnet.threads.count | 11 ± (11 - 11) | 11 ± (11 - 11) | +0.0% | ✅ |
| .NET 6 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 254.93 ± (251.33 - 258.54) ms | 1289.99 ± (1288.21 - 1291.77) ms | +406.0% | ✅⬆️ |
| process.time_to_main_ms | 482.71 ± (482.02 - 483.41) ms | 477.62 ± (476.55 - 478.70) ms | -1.1% | ✅ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 18 ± (18 - 18) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 48.40 ± (48.38 - 48.42) MB | 50.02 ± (49.99 - 50.05) MB | +3.3% | ✅⬆️ |
| runtime.dotnet.threads.count | 28 ± (28 - 28) | 28 ± (28 - 28) | +0.0% | ✅⬆️ |
| .NET 8 - Baseline | ||||
| process.internal_duration_ms | 20.25 ± (20.21 - 20.29) ms | 19.81 ± (19.77 - 19.85) ms | -2.2% | ✅ |
| process.time_to_main_ms | 76.87 ± (76.71 - 77.03) ms | 74.63 ± (74.46 - 74.80) ms | -2.9% | ✅ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 7.68 ± (7.68 - 7.69) MB | 7.69 ± (7.68 - 7.70) MB | +0.1% | ✅⬆️ |
| runtime.dotnet.threads.count | 10 ± (10 - 10) | 10 ± (10 - 10) | +0.0% | ✅ |
| .NET 8 - Bailout | ||||
| process.internal_duration_ms | 20.15 ± (20.10 - 20.21) ms | 19.90 ± (19.86 - 19.94) ms | -1.3% | ✅ |
| process.time_to_main_ms | 77.50 ± (77.32 - 77.67) ms | 76.07 ± (75.88 - 76.25) ms | -1.8% | ✅ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 7.74 ± (7.73 - 7.75) MB | 7.72 ± (7.71 - 7.72) MB | -0.2% | ✅ |
| runtime.dotnet.threads.count | 11 ± (11 - 11) | 11 ± (11 - 11) | +0.0% | ✅ |
| .NET 8 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 193.52 ± (192.88 - 194.16) ms | 1218.21 ± (1216.42 - 1219.99) ms | +529.5% | ✅⬆️ |
| process.time_to_main_ms | 462.15 ± (461.54 - 462.76) ms | 458.26 ± (457.02 - 459.50) ms | -0.8% | ✅ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 18 ± (18 - 18) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 36.15 ± (36.11 - 36.18) MB | 37.49 ± (37.46 - 37.51) MB | +3.7% | ✅⬆️ |
| runtime.dotnet.threads.count | 27 ± (27 - 27) | 27 ± (26 - 27) | -0.4% | ✅ |
HttpMessageHandler
| Metric | Master (Mean ± 95% CI) | Current (Mean ± 95% CI) | Change | Status |
|---|---|---|---|---|
| .NET Framework 4.8 - Baseline | ||||
| duration | 219.33 ± (219.40 - 220.80) ms | 214.65 ± (214.81 - 216.33) ms | -2.1% | ✅ |
| .NET Framework 4.8 - Bailout | ||||
| duration | 224.47 ± (224.39 - 225.88) ms | 219.57 ± (220.38 - 222.57) ms | -2.2% | ✅ |
| .NET Framework 4.8 - CallTarget+Inlining+NGEN | ||||
| duration | 1247.22 ± (1247.00 - 1255.89) ms | 1226.94 ± (1228.06 - 1235.65) ms | -1.6% | ✅ |
| .NET Core 3.1 - Baseline | ||||
| process.internal_duration_ms | 213.38 ± (212.79 - 213.96) ms | 207.44 ± (206.84 - 208.05) ms | -2.8% | ✅ |
| process.time_to_main_ms | 92.32 ± (92.02 - 92.63) ms | 89.90 ± (89.64 - 90.17) ms | -2.6% | ✅ |
| runtime.dotnet.exceptions.count | 3 ± (3 - 3) | 3 ± (3 - 3) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 15.88 ± (15.86 - 15.90) MB | 15.90 ± (15.89 - 15.92) MB | +0.1% | ✅⬆️ |
| runtime.dotnet.threads.count | 20 ± (20 - 20) | 20 ± (20 - 20) | -0.8% | ✅ |
| .NET Core 3.1 - Bailout | ||||
| process.internal_duration_ms | 213.72 ± (213.11 - 214.33) ms | 208.37 ± (207.80 - 208.94) ms | -2.5% | ✅ |
| process.time_to_main_ms | 94.54 ± (94.26 - 94.82) ms | 91.11 ± (90.88 - 91.33) ms | -3.6% | ✅ |
| runtime.dotnet.exceptions.count | 3 ± (3 - 3) | 3 ± (3 - 3) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 15.92 ± (15.90 - 15.93) MB | 15.89 ± (15.88 - 15.91) MB | -0.2% | ✅ |
| runtime.dotnet.threads.count | 21 ± (21 - 21) | 21 ± (21 - 21) | -0.2% | ✅ |
| .NET Core 3.1 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 443.28 ± (439.66 - 446.91) ms | 445.91 ± (441.82 - 450.00) ms | +0.6% | ✅⬆️ |
| process.time_to_main_ms | 527.40 ± (526.25 - 528.55) ms | 517.82 ± (516.99 - 518.65) ms | -1.8% | ✅ |
| runtime.dotnet.exceptions.count | 3 ± (3 - 3) | 3 ± (3 - 3) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 58.06 ± (57.92 - 58.19) MB | 58.03 ± (57.89 - 58.17) MB | -0.1% | ✅ |
| runtime.dotnet.threads.count | 30 ± (30 - 30) | 30 ± (30 - 30) | -0.5% | ✅ |
| .NET 6 - Baseline | ||||
| process.internal_duration_ms | 220.30 ± (219.42 - 221.19) ms | 212.71 ± (212.09 - 213.32) ms | -3.4% | ✅ |
| process.time_to_main_ms | 80.81 ± (80.52 - 81.10) ms | 77.63 ± (77.43 - 77.82) ms | -3.9% | ✅ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 16.13 ± (16.11 - 16.14) MB | 16.22 ± (16.20 - 16.23) MB | +0.6% | ✅⬆️ |
| runtime.dotnet.threads.count | 20 ± (19 - 20) | 19 ± (19 - 19) | -1.0% | ✅ |
| .NET 6 - Bailout | ||||
| process.internal_duration_ms | 225.40 ± (224.03 - 226.76) ms | 212.99 ± (212.34 - 213.63) ms | -5.5% | ✅ |
| process.time_to_main_ms | 82.70 ± (82.39 - 83.01) ms | 79.23 ± (78.98 - 79.48) ms | -4.2% | ✅ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 16.14 ± (16.12 - 16.15) MB | 16.24 ± (16.22 - 16.26) MB | +0.6% | ✅⬆️ |
| runtime.dotnet.threads.count | 20 ± (20 - 21) | 20 ± (20 - 21) | +0.0% | ✅⬆️ |
| .NET 6 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 478.46 ± (473.92 - 482.99) ms | 1550.64 ± (1543.90 - 1557.37) ms | +224.1% | ✅⬆️ |
| process.time_to_main_ms | 496.51 ± (495.42 - 497.59) ms | 488.97 ± (488.11 - 489.83) ms | -1.5% | ✅ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 22 ± (22 - 22) | +450.0% | ✅⬆️ |
| runtime.dotnet.mem.committed | 57.52 ± (57.36 - 57.69) MB | 63.02 ± (62.97 - 63.07) MB | +9.6% | ❌⬆️ |
| runtime.dotnet.threads.count | 30 ± (30 - 30) | 30 ± (30 - 30) | +1.6% | ✅⬆️ |
| .NET 8 - Baseline | ||||
| process.internal_duration_ms | 218.07 ± (217.25 - 218.88) ms | 211.73 ± (211.08 - 212.38) ms | -2.9% | ✅ |
| process.time_to_main_ms | 79.17 ± (78.91 - 79.43) ms | 76.78 ± (76.59 - 76.98) ms | -3.0% | ✅ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 11.47 ± (11.46 - 11.49) MB | 11.55 ± (11.54 - 11.57) MB | +0.7% | ✅⬆️ |
| runtime.dotnet.threads.count | 19 ± (19 - 19) | 19 ± (19 - 19) | -0.0% | ✅ |
| .NET 8 - Bailout | ||||
| process.internal_duration_ms | 217.19 ± (216.45 - 217.93) ms | 211.12 ± (210.44 - 211.80) ms | -2.8% | ✅ |
| process.time_to_main_ms | 80.21 ± (79.97 - 80.45) ms | 77.97 ± (77.76 - 78.17) ms | -2.8% | ✅ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 11.51 ± (11.49 - 11.52) MB | 11.60 ± (11.58 - 11.61) MB | +0.8% | ✅⬆️ |
| runtime.dotnet.threads.count | 20 ± (20 - 20) | 20 ± (20 - 20) | -0.1% | ✅ |
| .NET 8 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 492.94 ± (488.16 - 497.73) ms | 1528.58 ± (1525.79 - 1531.36) ms | +210.1% | ✅⬆️ |
| process.time_to_main_ms | 480.67 ± (479.84 - 481.50) ms | 472.40 ± (471.16 - 473.65) ms | -1.7% | ✅ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 22 ± (22 - 22) | +450.0% | ✅⬆️ |
| runtime.dotnet.mem.committed | 49.80 ± (49.77 - 49.84) MB | 54.38 ± (54.34 - 54.41) MB | +9.2% | ✅⬆️ |
| runtime.dotnet.threads.count | 29 ± (29 - 29) | 29 ± (29 - 29) | +0.7% | ✅⬆️ |
Comparison explanation
Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
- Welch test with statistical test for significance of 5%
- Only results indicating a difference greater than 5% and 5 ms are considered.
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.
Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).
Duration charts
FakeDbCommand (.NET Framework 4.8)
gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8211) - mean (75ms) : 73, 77
master - mean (76ms) : 73, 79
section Bailout
This PR (8211) - mean (80ms) : 78, 82
master - mean (80ms) : 78, 83
section CallTarget+Inlining+NGEN
This PR (8211) - mean (1,097ms) : 1042, 1153
master - mean (1,099ms) : 1050, 1148
FakeDbCommand (.NET Core 3.1)
gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8211) - mean (117ms) : 114, 120
master - mean (119ms) : 115, 123
section Bailout
This PR (8211) - mean (118ms) : 115, 121
master - mean (121ms) : 117, 124
section CallTarget+Inlining+NGEN
This PR (8211) - mean (776ms) : 716, 835
master - mean (779ms) : 720, 838
FakeDbCommand (.NET 6)
gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8211) - mean (104ms) : 100, 107
master - mean (107ms) : 104, 110
section Bailout
This PR (8211) - mean (105ms) : 102, 107
master - mean (107ms) : 105, 110
section CallTarget+Inlining+NGEN
This PR (8211) - mean (1,801ms) : crit, 1782, 1820
master - mean (771ms) : 706, 836
FakeDbCommand (.NET 8)
gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8211) - mean (102ms) : 99, 106
master - mean (105ms) : 102, 108
section Bailout
This PR (8211) - mean (104ms) : 101, 107
master - mean (106ms) : 103, 108
section CallTarget+Inlining+NGEN
This PR (8211) - mean (1,714ms) : crit, 1682, 1746
master - mean (688ms) : 663, 714
HttpMessageHandler (.NET Framework 4.8)
gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8211) - mean (216ms) : 204, 227
master - mean (220ms) : 210, 231
section Bailout
This PR (8211) - mean (221ms) : 205, 238
master - mean (225ms) : 214, 236
section CallTarget+Inlining+NGEN
This PR (8211) - mean (1,232ms) : 1175, 1289
master - mean (1,251ms) : 1183, 1320
HttpMessageHandler (.NET Core 3.1)
gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8211) - mean (307ms) : 296, 319
master - mean (316ms) : 305, 327
section Bailout
This PR (8211) - mean (309ms) : 297, 322
master - mean (318ms) : 306, 331
section CallTarget+Inlining+NGEN
This PR (8211) - mean (1,000ms) : 930, 1070
master - mean (1,015ms) : 958, 1071
HttpMessageHandler (.NET 6)
gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8211) - mean (300ms) : 289, 311
master - mean (311ms) : 294, 328
section Bailout
This PR (8211) - mean (302ms) : 287, 316
master - mean (318ms) : 293, 343
section CallTarget+Inlining+NGEN
This PR (8211) - mean (2,076ms) : crit, 1976, 2175
master - mean (1,017ms) : 943, 1091
HttpMessageHandler (.NET 8)
gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8211) - mean (299ms) : 285, 314
master - mean (308ms) : 295, 322
section Bailout
This PR (8211) - mean (300ms) : 286, 314
master - mean (309ms) : 295, 323
section CallTarget+Inlining+NGEN
This PR (8211) - mean (2,038ms) : crit, 2001, 2075
master - mean (1,010ms) : 939, 1082
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e729239afd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| AgentUri = traceSettings.AgentUri; | ||
|
|
||
| var otlpTraceSettings = GetOtlpTracesTransport(signalEndpoint: rawSettings.OtlpTracesEndpoint, generalEndpoint: rawSettings.OtlpEndpoint, signalProtocol: rawSettings.OtlpTracesProtocol, generalProtocol: rawSettings.OtlpProtocol); | ||
| TracesEncoding = otlpTraceSettings.Encoding; |
There was a problem hiding this comment.
Gate OTLP transport behind OTEL_TRACES_EXPORTER
TracesEncoding is unconditionally overwritten from GetOtlpTracesTransport(...), so on .NET 6+ the tracer switches to ManagedApiOtlp even when OTEL_TRACES_EXPORTER is not set. This changes the default behavior from Datadog agent transport to OTLP defaults (localhost:4318), which can silently break trace delivery for existing deployments that only run the Datadog agent.
Useful? React with 👍 / 👎.
| } | ||
| else if (protocol != OtlpProtocol.Grpc) | ||
| { | ||
| if (signalEndpoint!.EndsWith("/") && TryGetAgentUriAndTransport($"{generalEndpoint!}{defaultHttpRelativePath}", origin, encoding, protocol, out var httpSettings)) |
There was a problem hiding this comment.
Handle OTLP base endpoint fallback without null dereference
In the generalEndpoint fallback path, the slash logic dereferences signalEndpoint!; if users configure only OTEL_EXPORTER_OTLP_ENDPOINT (without a signal-specific endpoint), this throws NullReferenceException during settings initialization. This makes a common OTLP configuration path fail before export starts.
Useful? React with 👍 / 👎.
| TracesEncoding.OtlpJson => new OtlpTracesJsonSerializer(), | ||
| _ => new SpanBufferMessagePackSerializer(SpanFormatterResolver.Instance), |
There was a problem hiding this comment.
Avoid MessagePack serialization for OTLP protobuf mode
When TracesEncoding is OtlpProtobuf (for grpc/http/protobuf protocol settings), this switch falls back to SpanBufferMessagePackSerializer instead of an OTLP serializer. That causes OTLP protobuf configurations to emit Datadog MessagePack payloads, which OTLP collectors will not parse correctly.
Useful? React with 👍 / 👎.
| _apiRequestFactory = apiRequestFactory; | ||
| _tracesEncoding = exporterSettings.TracesEncoding; | ||
| _tracesEndpoint = exporterSettings.OtlpTracesEndpoint; | ||
| _statsEndpoint = exporterSettings.OtlpMetricsEndpoint; |
There was a problem hiding this comment.
Apply OTLP trace headers and timeout in ApiOtlp
The OTLP trace config keys for OTEL_EXPORTER_OTLP_TRACES_HEADERS and OTEL_EXPORTER_OTLP_TRACES_TIMEOUT are parsed, but ApiOtlp only consumes encoding and endpoints here and never wires trace headers/timeout into outgoing trace requests. In environments where the collector requires auth headers or tuned timeouts, trace export will fail despite the documented settings.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e729239afd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| AgentUri = traceSettings.AgentUri; | ||
|
|
||
| var otlpTraceSettings = GetOtlpTracesTransport(signalEndpoint: rawSettings.OtlpTracesEndpoint, generalEndpoint: rawSettings.OtlpEndpoint, signalProtocol: rawSettings.OtlpTracesProtocol, generalProtocol: rawSettings.OtlpProtocol); | ||
| TracesEncoding = otlpTraceSettings.Encoding; |
There was a problem hiding this comment.
Gate OTLP trace encoding behind OTEL_TRACES_EXPORTER
This constructor always overwrites TracesEncoding with GetOtlpTracesTransport(...) regardless of OtelTracesExporter, so when OTEL_TRACES_EXPORTER is unset the default OTLP trace encoding still gets selected and .NET 6+ flows into ManagedApiOtlp instead of the Datadog trace agent path. That changes the default transport behavior for all users on supported runtimes, even though OTLP export is documented as opt-in.
Useful? React with 👍 / 👎.
| } | ||
| else if (protocol != OtlpProtocol.Grpc) | ||
| { | ||
| if (signalEndpoint!.EndsWith("/") && TryGetAgentUriAndTransport($"{generalEndpoint!}{defaultHttpRelativePath}", origin, encoding, protocol, out var httpSettings)) |
There was a problem hiding this comment.
Use general endpoint safely when deriving OTLP HTTP URL
When only OTEL_EXPORTER_OTLP_ENDPOINT is configured (without signal-specific endpoint), this branch dereferences signalEndpoint! to decide slash handling, causing a NullReferenceException during settings initialization. It also checks the wrong variable for trailing slash, so URL composition can be incorrect even when both endpoints are present.
Useful? React with 👍 / 👎.
| } | ||
|
|
||
| if (metric.SnapshotCount > 0) | ||
| if (metric.SnapshotCount > 0 & metric.SnapshotMin != double.NaN) |
There was a problem hiding this comment.
Skip NaN histogram min/max when serializing OTLP metrics
The NaN guard here is ineffective because x != double.NaN is always true in .NET, so min/max are still serialized whenever SnapshotCount > 0. In this commit, OTLP trace-metric points set SnapshotMin/SnapshotMax to NaN, so the exporter emits NaN min/max fields instead of omitting them, which can break downstream handling of histogram points.
Useful? React with 👍 / 👎.
| OTEL_EXPORTER_OTLP_TRACES_PROTOCOL: | | ||
| Configuration key to set the OTLP protocol for traces export. | ||
| Takes precedence over <see cref="ExporterOtlpProtocol"/>. | ||
| Valid values: http/json, defaults to http/json. |
There was a problem hiding this comment.
should the valid values be http/json, http/protobuf and grpc?
There was a problem hiding this comment.
Eventually yes, but right now I've only implemented http/json so I haven't included http/protobuf or grpc
Summary of changes
Adds experimental support for exporting traces using OTLP rather than the Datadog MessagePack protocols. This allows the DD SDK to send traces (and trace metrics) to an OTel collector rather a Datadog Trace Agent, with limited support for non-APM products.
This feature is enabled by setting
OTEL_TRACES_EXPORTER=otlp.Note: This feature is currently only supported for .NET 6+. In addition to sending sampled traces to the OTLP collector, we must also send trace metrics to the OTLP collector to power the full Datadog APM experience. To do this, we use our built-in OTLP Metrics support which is limited to our .NET 6+ build.
Configuration
OTEL_TRACES_EXPORTER=otlpOTEL_EXPORTER_OTLP_TRACES_ENDPOINTOTEL_EXPORTER_OTLP_TRACES_HEADERSOTEL_EXPORTER_OTLP_TRACES_PROTOCOLOTEL_EXPORTER_OTLP_TRACES_TIMEOUTReason for change
We are seeing an increasing number of scenarios were users have applications instrumented with the OTel SDK sending data to OTel collectors, and they would like to get additional features offered by the DD SDK without needing to update their OTel collector deployments. Although there will be follow-up work, this provides the ability for users to write vendor-neutral API instrumentation and emit vendor-neutral telemetry data so DD SDK users don't have to feel locked in when setting up Datadog APM.
Implementation details
ExporterSettingsclass and add them as properties there, so that we have the ability to refresh the exporters if there are changes to the configurations.Datadog.Trace.Agent.TracesEncodingenum to indicate which Datadog Message protocol or OTLP protocol we are using.ISpanBufferSerializerinterface to abstract away the MessagePack and OTLP serialization implementations from theSpanBufferclass. Notably, to make the OTLP serialization work with our pre-emptive encoding of spans into theSpanBuffer, add aFinishBodymethod so that we can create valid JSON after appending spans multiple times to the theresource_spans[0].scope_spans[0].spansarray.OtlpTracesJsonSerializerimplementation which can serialize theTraceChunkModel(and itsSpanobjects) to its corresponding OTLP JSON structure.OtlpMapperstatic class. This class also contains the logic for converting the contents of aStatsBuffer(and itsDDSketchobjects) to its corresponding OTLP Histogram. The specific names and attributes needed for the OTLP Histogram to be recognized for APM trace metrics is in the process of being defined and may change.DD_TRACE_STATS_COMPUTATION_ENABLED=true) while also updating theStatsAggregationKeyso we have distinct histograms for errors and top-level spans.SpanBufferby theOtlpTracesJsonSerializer, theApiOtlpclass will receive them in theIApi.SendTracesAsyncmethod and handle the HTTP request logic. This includes handling status codes as defined by the OTLP specification.ApiOtlpclass receives the currentStatsBufferin theIApi.SendStatsAsyncmethod, it calls into theOtlpMapperclass to convert it into its corresponding OTLP Histogram then hands this off to the pre-existingOtlpExporterfrom our built-in OTel Metrics support to handle the HTTP request logic.Test coverage
This adds a new integration test
OpenTelemetrySdkTests.SubmitsOtlpTracesthat sends OTLP traces to the dd-apm-test-agent, retrieves the payload, normalizes it, and does snapshot testing.Other details
Since this is a first-pass and experimental implementation, there is some additional work needed to polish the feature for general use. That includes the following:
OpenTelemetrySdkTests.SubmitsOtlpTracestest, assert against the trace metrics (sent with OTLP)TracerSettingsnow that they are being read inExporterSettings