Remote configuration tests and performance improvements#8237
Remote configuration tests and performance improvements#8237andrewlock wants to merge 2 commits intoandrew/moar-json-array-poolfrom
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2a6bccf061
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // Verify all characters between are digits | ||
| for (var i = digitStart; i < secondSlash; i++) | ||
| { | ||
| if (path[i] is < '0' or > '9') |
There was a problem hiding this comment.
Preserve Unicode digit handling for datadog org segment
The new parser narrows org-id validation to ASCII ('0'..'9'), but the previous implementation used \d in .NET regex, which accepts all Unicode decimal digits. That is a behavioral regression: paths like datadog/١/PRODUCT/id/file used to parse and now throw, which causes RcmSubscriptionManager.ProcessResponse to fail that poll and skip applying configs whenever a non-ASCII decimal digit appears in the org segment.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
I'm like 99% sure that was probably a bug, but we should check...
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (8237) and master.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Metric | Master (Mean ± 95% CI) | Current (Mean ± 95% CI) | Change | Status |
|---|---|---|---|---|
| .NET Framework 4.8 - Bailout | ||||
| duration | 194.61 ± (194.58 - 195.07) ms | 206.18 ± (205.37 - 206.50) ms | +5.9% | ❌⬆️ |
Full Metrics Comparison
FakeDbCommand
| Metric | Master (Mean ± 95% CI) | Current (Mean ± 95% CI) | Change | Status |
|---|---|---|---|---|
| .NET Framework 4.8 - Baseline | ||||
| duration | 73.60 ± (73.63 - 73.95) ms | 76.33 ± (76.33 - 76.65) ms | +3.7% | ✅⬆️ |
| .NET Framework 4.8 - Bailout | ||||
| duration | 77.86 ± (77.86 - 78.19) ms | 81.15 ± (80.97 - 81.29) ms | +4.2% | ✅⬆️ |
| .NET Framework 4.8 - CallTarget+Inlining+NGEN | ||||
| duration | 1067.52 ± (1068.28 - 1074.61) ms | 1094.93 ± (1095.93 - 1101.41) ms | +2.6% | ✅⬆️ |
| .NET Core 3.1 - Baseline | ||||
| process.internal_duration_ms | 22.41 ± (22.37 - 22.45) ms | 23.33 ± (23.27 - 23.39) ms | +4.1% | ✅⬆️ |
| process.time_to_main_ms | 84.61 ± (84.42 - 84.79) ms | 89.23 ± (89.03 - 89.42) ms | +5.5% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.92 ± (10.91 - 10.92) MB | 10.93 ± (10.92 - 10.93) MB | +0.1% | ✅⬆️ |
| runtime.dotnet.threads.count | 12 ± (12 - 12) | 12 ± (12 - 12) | +0.0% | ✅ |
| .NET Core 3.1 - Bailout | ||||
| process.internal_duration_ms | 22.38 ± (22.34 - 22.42) ms | 23.26 ± (23.21 - 23.32) ms | +4.0% | ✅⬆️ |
| process.time_to_main_ms | 86.01 ± (85.83 - 86.19) ms | 90.60 ± (90.37 - 90.82) ms | +5.3% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.96 ± (10.95 - 10.96) MB | 10.91 ± (10.91 - 10.92) MB | -0.4% | ✅ |
| runtime.dotnet.threads.count | 13 ± (13 - 13) | 13 ± (13 - 13) | +0.0% | ✅ |
| .NET Core 3.1 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 240.44 ± (236.44 - 244.45) ms | 236.09 ± (232.18 - 240.01) ms | -1.8% | ✅ |
| process.time_to_main_ms | 485.58 ± (485.00 - 486.15) ms | 504.10 ± (503.33 - 504.87) ms | +3.8% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 47.64 ± (47.62 - 47.66) MB | 47.62 ± (47.60 - 47.64) MB | -0.1% | ✅ |
| runtime.dotnet.threads.count | 28 ± (28 - 28) | 28 ± (28 - 28) | +0.0% | ✅ |
| .NET 6 - Baseline | ||||
| process.internal_duration_ms | 21.33 ± (21.28 - 21.38) ms | 22.18 ± (22.13 - 22.23) ms | +4.0% | ✅⬆️ |
| process.time_to_main_ms | 73.84 ± (73.69 - 73.99) ms | 77.82 ± (77.65 - 77.98) ms | +5.4% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.63 ± (10.63 - 10.64) MB | 10.63 ± (10.63 - 10.64) MB | -0.0% | ✅ |
| runtime.dotnet.threads.count | 10 ± (10 - 10) | 10 ± (10 - 10) | +0.0% | ✅ |
| .NET 6 - Bailout | ||||
| process.internal_duration_ms | 21.25 ± (21.20 - 21.31) ms | 22.07 ± (22.01 - 22.14) ms | +3.9% | ✅⬆️ |
| process.time_to_main_ms | 75.06 ± (74.90 - 75.23) ms | 78.72 ± (78.57 - 78.87) ms | +4.9% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.69 ± (10.68 - 10.69) MB | 10.71 ± (10.71 - 10.72) MB | +0.3% | ✅⬆️ |
| runtime.dotnet.threads.count | 11 ± (11 - 11) | 11 ± (11 - 11) | +0.0% | ✅ |
| .NET 6 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 255.98 ± (253.38 - 258.59) ms | 257.38 ± (253.75 - 261.02) ms | +0.5% | ✅⬆️ |
| process.time_to_main_ms | 466.93 ± (466.39 - 467.47) ms | 485.09 ± (484.44 - 485.74) ms | +3.9% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 48.41 ± (48.39 - 48.44) MB | 48.29 ± (48.27 - 48.31) MB | -0.3% | ✅ |
| runtime.dotnet.threads.count | 28 ± (28 - 28) | 28 ± (28 - 28) | +0.2% | ✅⬆️ |
| .NET 8 - Baseline | ||||
| process.internal_duration_ms | 19.58 ± (19.55 - 19.61) ms | 20.20 ± (20.15 - 20.25) ms | +3.1% | ✅⬆️ |
| process.time_to_main_ms | 73.54 ± (73.38 - 73.71) ms | 76.82 ± (76.62 - 77.01) ms | +4.5% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 7.66 ± (7.66 - 7.67) MB | 7.67 ± (7.66 - 7.67) MB | +0.1% | ✅⬆️ |
| runtime.dotnet.threads.count | 10 ± (10 - 10) | 10 ± (10 - 10) | +0.0% | ✅ |
| .NET 8 - Bailout | ||||
| process.internal_duration_ms | 19.65 ± (19.61 - 19.70) ms | 20.16 ± (20.12 - 20.21) ms | +2.6% | ✅⬆️ |
| process.time_to_main_ms | 74.82 ± (74.66 - 74.97) ms | 78.22 ± (78.03 - 78.40) ms | +4.5% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 7.71 ± (7.71 - 7.72) MB | 7.72 ± (7.71 - 7.73) MB | +0.0% | ✅⬆️ |
| runtime.dotnet.threads.count | 11 ± (11 - 11) | 11 ± (11 - 11) | +0.0% | ✅ |
| .NET 8 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 189.13 ± (188.38 - 189.88) ms | 193.33 ± (192.46 - 194.21) ms | +2.2% | ✅⬆️ |
| process.time_to_main_ms | 451.15 ± (450.51 - 451.78) ms | 464.58 ± (463.93 - 465.23) ms | +3.0% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 36.01 ± (35.97 - 36.04) MB | 36.14 ± (36.10 - 36.19) MB | +0.4% | ✅⬆️ |
| runtime.dotnet.threads.count | 27 ± (27 - 27) | 27 ± (27 - 27) | -0.1% | ✅ |
HttpMessageHandler
| Metric | Master (Mean ± 95% CI) | Current (Mean ± 95% CI) | Change | Status |
|---|---|---|---|---|
| .NET Framework 4.8 - Baseline | ||||
| duration | 191.74 ± (191.76 - 192.80) ms | 198.99 ± (198.69 - 199.94) ms | +3.8% | ✅⬆️ |
| .NET Framework 4.8 - Bailout | ||||
| duration | 194.61 ± (194.58 - 195.07) ms | 206.18 ± (205.37 - 206.50) ms | +5.9% | ❌⬆️ |
| .NET Framework 4.8 - CallTarget+Inlining+NGEN | ||||
| duration | 1133.00 ± (1135.30 - 1142.49) ms | 1173.04 ± (1171.89 - 1181.03) ms | +3.5% | ✅⬆️ |
| .NET Core 3.1 - Baseline | ||||
| process.internal_duration_ms | 186.45 ± (186.12 - 186.77) ms | 194.41 ± (193.86 - 194.96) ms | +4.3% | ✅⬆️ |
| process.time_to_main_ms | 80.48 ± (80.27 - 80.70) ms | 84.30 ± (83.99 - 84.60) ms | +4.7% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 3 ± (3 - 3) | 3 ± (3 - 3) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 16.21 ± (16.17 - 16.24) MB | 16.05 ± (16.04 - 16.07) MB | -1.0% | ✅ |
| runtime.dotnet.threads.count | 19 ± (19 - 20) | 20 ± (19 - 20) | +0.2% | ✅⬆️ |
| .NET Core 3.1 - Bailout | ||||
| process.internal_duration_ms | 185.31 ± (184.98 - 185.63) ms | 194.71 ± (194.20 - 195.21) ms | +5.1% | ✅⬆️ |
| process.time_to_main_ms | 81.90 ± (81.73 - 82.07) ms | 85.93 ± (85.65 - 86.20) ms | +4.9% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 3 ± (3 - 3) | 3 ± (3 - 3) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 16.28 ± (16.25 - 16.31) MB | 16.09 ± (16.07 - 16.10) MB | -1.2% | ✅ |
| runtime.dotnet.threads.count | 21 ± (21 - 21) | 21 ± (20 - 21) | -0.6% | ✅ |
| .NET Core 3.1 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 434.52 ± (432.09 - 436.95) ms | 440.54 ± (437.70 - 443.37) ms | +1.4% | ✅⬆️ |
| process.time_to_main_ms | 470.33 ± (469.83 - 470.83) ms | 489.86 ± (488.60 - 491.11) ms | +4.2% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 3 ± (3 - 3) | 3 ± (3 - 3) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 57.94 ± (57.82 - 58.06) MB | 58.10 ± (57.98 - 58.22) MB | +0.3% | ✅⬆️ |
| runtime.dotnet.threads.count | 29 ± (29 - 29) | 29 ± (29 - 30) | +0.1% | ✅⬆️ |
| .NET 6 - Baseline | ||||
| process.internal_duration_ms | 190.39 ± (190.06 - 190.72) ms | 198.80 ± (198.35 - 199.24) ms | +4.4% | ✅⬆️ |
| process.time_to_main_ms | 69.89 ± (69.69 - 70.08) ms | 73.16 ± (72.93 - 73.39) ms | +4.7% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 16.15 ± (16.02 - 16.29) MB | 16.31 ± (16.29 - 16.33) MB | +1.0% | ✅⬆️ |
| runtime.dotnet.threads.count | 18 ± (18 - 18) | 19 ± (19 - 19) | +5.9% | ✅⬆️ |
| .NET 6 - Bailout | ||||
| process.internal_duration_ms | 188.91 ± (188.65 - 189.17) ms | 193.27 ± (192.83 - 193.70) ms | +2.3% | ✅⬆️ |
| process.time_to_main_ms | 70.64 ± (70.54 - 70.75) ms | 72.42 ± (72.25 - 72.59) ms | +2.5% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 15.82 ± (15.65 - 15.99) MB | 16.37 ± (16.34 - 16.41) MB | +3.5% | ✅⬆️ |
| runtime.dotnet.threads.count | 19 ± (19 - 19) | 20 ± (20 - 20) | +6.4% | ✅⬆️ |
| .NET 6 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 448.62 ± (446.97 - 450.28) ms | 460.04 ± (457.52 - 462.56) ms | +2.5% | ✅⬆️ |
| process.time_to_main_ms | 446.89 ± (446.38 - 447.41) ms | 465.07 ± (464.04 - 466.11) ms | +4.1% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 58.15 ± (58.04 - 58.26) MB | 57.68 ± (57.55 - 57.82) MB | -0.8% | ✅ |
| runtime.dotnet.threads.count | 29 ± (29 - 29) | 29 ± (29 - 29) | +0.1% | ✅⬆️ |
| .NET 8 - Baseline | ||||
| process.internal_duration_ms | 187.68 ± (187.40 - 187.96) ms | 192.29 ± (191.85 - 192.73) ms | +2.5% | ✅⬆️ |
| process.time_to_main_ms | 69.19 ± (69.02 - 69.37) ms | 70.92 ± (70.66 - 71.17) ms | +2.5% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 11.76 ± (11.73 - 11.79) MB | 11.70 ± (11.68 - 11.72) MB | -0.5% | ✅ |
| runtime.dotnet.threads.count | 18 ± (18 - 18) | 18 ± (18 - 18) | +0.9% | ✅⬆️ |
| .NET 8 - Bailout | ||||
| process.internal_duration_ms | 187.45 ± (187.15 - 187.76) ms | 191.23 ± (190.82 - 191.63) ms | +2.0% | ✅⬆️ |
| process.time_to_main_ms | 70.34 ± (70.20 - 70.48) ms | 71.79 ± (71.64 - 71.93) ms | +2.1% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 11.77 ± (11.69 - 11.85) MB | 11.80 ± (11.77 - 11.83) MB | +0.3% | ✅⬆️ |
| runtime.dotnet.threads.count | 19 ± (18 - 19) | 19 ± (19 - 19) | +3.8% | ✅⬆️ |
| .NET 8 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 363.28 ± (361.91 - 364.66) ms | 370.83 ± (369.41 - 372.26) ms | +2.1% | ✅⬆️ |
| process.time_to_main_ms | 429.75 ± (429.20 - 430.29) ms | 441.38 ± (440.38 - 442.37) ms | +2.7% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 47.73 ± (47.68 - 47.78) MB | 47.74 ± (47.71 - 47.78) MB | +0.0% | ✅⬆️ |
| runtime.dotnet.threads.count | 29 ± (29 - 29) | 29 ± (29 - 29) | +0.1% | ✅⬆️ |
Comparison explanation
Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
- Welch test with statistical test for significance of 5%
- Only results indicating a difference greater than 5% and 5 ms are considered.
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.
Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).
Duration charts
FakeDbCommand (.NET Framework 4.8)
gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8237) - mean (76ms) : 74, 79
master - mean (74ms) : 72, 76
section Bailout
This PR (8237) - mean (81ms) : 80, 83
master - mean (78ms) : 76, 80
section CallTarget+Inlining+NGEN
This PR (8237) - mean (1,099ms) : 1060, 1138
master - mean (1,071ms) : 1026, 1117
FakeDbCommand (.NET Core 3.1)
gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8237) - mean (120ms) : 117, 123
master - mean (114ms) : 111, 117
section Bailout
This PR (8237) - mean (121ms) : 119, 124
master - mean (115ms) : 113, 117
section CallTarget+Inlining+NGEN
This PR (8237) - mean (777ms) : 713, 842
master - mean (757ms) : 693, 822
FakeDbCommand (.NET 6)
gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8237) - mean (107ms) : 104, 110
master - mean (102ms) : 98, 105
section Bailout
This PR (8237) - mean (107ms) : 105, 110
master - mean (103ms) : 101, 105
section CallTarget+Inlining+NGEN
This PR (8237) - mean (774ms) : 710, 839
master - mean (759ms) : 712, 805
FakeDbCommand (.NET 8)
gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8237) - mean (105ms) : 102, 108
master - mean (101ms) : 98, 104
section Bailout
This PR (8237) - mean (106ms) : 104, 108
master - mean (102ms) : 100, 105
section CallTarget+Inlining+NGEN
This PR (8237) - mean (687ms) : 664, 710
master - mean (672ms) : 653, 692
HttpMessageHandler (.NET Framework 4.8)
gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8237) - mean (199ms) : 190, 209
master - mean (192ms) : 187, 198
section Bailout
This PR (8237) - mean (206ms) : crit, 198, 214
master - mean (195ms) : 192, 197
section CallTarget+Inlining+NGEN
This PR (8237) - mean (1,176ms) : 1108, 1245
master - mean (1,139ms) : 1087, 1191
HttpMessageHandler (.NET Core 3.1)
gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8237) - mean (288ms) : 273, 302
master - mean (275ms) : 270, 280
section Bailout
This PR (8237) - mean (289ms) : 278, 301
master - mean (275ms) : 271, 280
section CallTarget+Inlining+NGEN
This PR (8237) - mean (961ms) : 924, 998
master - mean (934ms) : 890, 978
HttpMessageHandler (.NET 6)
gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8237) - mean (281ms) : 270, 291
master - mean (268ms) : 264, 273
section Bailout
This PR (8237) - mean (274ms) : 268, 280
master - mean (268ms) : 264, 271
section CallTarget+Inlining+NGEN
This PR (8237) - mean (955ms) : 912, 998
master - mean (926ms) : 907, 945
HttpMessageHandler (.NET 8)
gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8237) - mean (273ms) : 266, 280
master - mean (267ms) : 262, 271
section Bailout
This PR (8237) - mean (273ms) : 267, 278
master - mean (267ms) : 262, 272
section CallTarget+Inlining+NGEN
This PR (8237) - mean (846ms) : 813, 879
master - mean (824ms) : 808, 841
Summary of changes
RcmSubscriptionManagerandRemoteConfigurationPathRemoteConfigurationPathReason for change
RemoteConfigurationPathis running aRegexon every file listed in the remote config response (which happens every 5s), but it's a simple pattern that can be easily directly implementedImplementation details
RcmSubscriptionManagerRemoteConfigurationPathcovering changes in this PRTest coverage
Unit tests in this PR cover compatibility with the existing implementation.
Simple benchmarking for the regex improvements:
Benchmarking code
Other details
https://datadoghq.atlassian.net/browse/LANGPLAT-940
All part of the Remote Config perf stack