Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI Visibility] Add support for MSTest 3.6.0 #6080

Merged
merged 6 commits into from
Sep 26, 2024
Merged

[CI Visibility] Add support for MSTest 3.6.0 #6080

merged 6 commits into from
Sep 26, 2024

Conversation

tonyredondo
Copy link
Member

@tonyredondo tonyredondo commented Sep 25, 2024 •
edited
Loading

Summary of changes

This PR adds support for MSTest 3.6.0

Reason for change

There is a new version of the NuGet package that we do not support.

Implementation details

Move the code that create the missing span to another integration that is covered by all supported mstest versions

Test coverage

All MSTest tests should pass, including the new version.

@tonyredondo tonyredondo self-assigned this Sep 25, 2024
@DataDog DataDog deleted a comment from datadog-ddstaging bot Sep 26, 2024
@DataDog DataDog deleted a comment from andrewlock Sep 26, 2024
@DataDog DataDog deleted a comment from andrewlock Sep 26, 2024
@DataDog DataDog deleted a comment from andrewlock Sep 26, 2024
@andrewlock
Copy link
Member

andrewlock commented Sep 26, 2024 •
edited
Loading

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6080) - mean (71ms)  : 68, 74
     .   : milestone, 71,
    master - mean (73ms)  : 66, 79
     .   : milestone, 73,

    section CallTarget+Inlining+NGEN
    This PR (6080) - mean (1,101ms)  : 1074, 1129
     .   : milestone, 1101,
    master - mean (1,109ms)  : 1080, 1138
     .   : milestone, 1109,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6080) - mean (110ms)  : 106, 114
     .   : milestone, 110,
    master - mean (109ms)  : 106, 113
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    This PR (6080) - mean (774ms)  : 757, 791
     .   : milestone, 774,
    master - mean (777ms)  : 753, 802
     .   : milestone, 777,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6080) - mean (94ms)  : 90, 97
     .   : milestone, 94,
    master - mean (94ms)  : 90, 99
     .   : milestone, 94,

    section CallTarget+Inlining+NGEN
    This PR (6080) - mean (728ms)  : 711, 746
     .   : milestone, 728,
    master - mean (734ms)  : 713, 755
     .   : milestone, 734,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6080) - mean (190ms)  : 187, 193
     .   : milestone, 190,
    master - mean (191ms)  : 188, 193
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (6080) - mean (1,196ms)  : 1169, 1222
     .   : milestone, 1196,
    master - mean (1,195ms)  : 1171, 1219
     .   : milestone, 1195,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6080) - mean (277ms)  : 270, 283
     .   : milestone, 277,
    master - mean (277ms)  : 273, 281
     .   : milestone, 277,

    section CallTarget+Inlining+NGEN
    This PR (6080) - mean (937ms)  : 919, 955
     .   : milestone, 937,
    master - mean (936ms)  : 921, 951
     .   : milestone, 936,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6080) - mean (266ms)  : 262, 269
     .   : milestone, 266,
    master - mean (264ms)  : 261, 268
     .   : milestone, 264,

    section CallTarget+Inlining+NGEN
    This PR (6080) - mean (925ms)  : 907, 944
     .   : milestone, 925,
    master - mean (931ms)  : 902, 960
     .   : milestone, 931,
Loading

@tonyredondo tonyredondo marked this pull request as ready for review September 26, 2024 14:37
@tonyredondo tonyredondo requested review from a team as code owners September 26, 2024 14:37
andrewlock
andrewlock approved these changes Sep 26, 2024
View reviewed changes
Copy link
Member

@andrewlock andrewlock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM :shipit:

@andrewlock
Copy link
Member

Benchmarks Report for tracer 🐌

Benchmarks for #6080 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.114
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 7.96μs 43.8ns 248ns 0.0156 0.00779 0 5.43 KB
master StartStopWithChild netcoreapp3.1 9.71μs 50.8ns 313ns 0.0186 0.00465 0 5.62 KB
master StartStopWithChild net472 16.1μs 51.2ns 198ns 1.03 0.315 0.105 6.06 KB
#6080 StartStopWithChild net6.0 7.94μs 44.7ns 319ns 0.0124 0.00414 0 5.42 KB
#6080 StartStopWithChild netcoreapp3.1 10.1μs 55.6ns 319ns 0.02 0.00998 0 5.62 KB
#6080 StartStopWithChild net472 16.3μs 80.1ns 367ns 1.01 0.293 0.0894 6.06 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 468μs 261ns 1.01μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 632μs 310ns 1.16μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 832μs 359ns 1.39μs 0.417 0 0 3.3 KB
#6080 WriteAndFlushEnrichedTraces net6.0 470μs 275ns 1.06μs 0 0 0 2.7 KB
#6080 WriteAndFlushEnrichedTraces netcoreapp3.1 634μs 329ns 1.27μs 0 0 0 2.7 KB
#6080 WriteAndFlushEnrichedTraces net472 833μs 588ns 2.2μs 0.414 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 195μs 1μs 4.6μs 0.188 0 0 18.45 KB
master SendRequest netcoreapp3.1 217μs 1.24μs 9.35μs 0.206 0 0 20.61 KB
master SendRequest net472 0.000514ns 0.000309ns 0.00111ns 0 0 0 0 b
#6080 SendRequest net6.0 195μs 1.12μs 8.86μs 0.192 0 0 18.45 KB
#6080 SendRequest netcoreapp3.1 214μs 1.15μs 6.58μs 0.214 0 0 20.61 KB
#6080 SendRequest net472 0.00311ns 0.00108ns 0.0042ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 569μs 2.68μs 12.3μs 0.551 0 0 41.64 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 672μs 3.05μs 11.8μs 0.327 0 0 41.67 KB
master WriteAndFlushEnrichedTraces net472 844μs 2.53μs 9.78μs 8.33 2.5 0.417 53.3 KB
#6080 WriteAndFlushEnrichedTraces net6.0 551μs 2.43μs 9.4μs 0.561 0 0 41.65 KB
#6080 WriteAndFlushEnrichedTraces netcoreapp3.1 659μs 2.4μs 9.31μs 0.327 0 0 41.78 KB
#6080 WriteAndFlushEnrichedTraces net472 852μs 3.2μs 12.4μs 8.08 2.55 0.425 53.24 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.31μs 1.61ns 6.22ns 0.0144 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 1.74μs 1.41ns 5.45ns 0.0138 0 0 1.02 KB
master ExecuteNonQuery net472 2.01μs 2.82ns 10.9ns 0.157 0 0 987 B
#6080 ExecuteNonQuery net6.0 1.29μs 1.03ns 3.97ns 0.0143 0 0 1.02 KB
#6080 ExecuteNonQuery netcoreapp3.1 1.75μs 1.02ns 3.82ns 0.0139 0 0 1.02 KB
#6080 ExecuteNonQuery net472 2.15μs 2.92ns 11.3ns 0.156 0 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6080

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync‑net6.0 1.114 1,232.76 1,106.29

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.24μs 0.259ns 0.969ns 0.0138 0 0 976 B
master CallElasticsearch netcoreapp3.1 1.59μs 0.595ns 2.3ns 0.0135 0 0 976 B
master CallElasticsearch net472 2.54μs 1.83ns 7.09ns 0.157 0 0 995 B
master CallElasticsearchAsync net6.0 1.23μs 0.705ns 2.73ns 0.0136 0 0 952 B
master CallElasticsearchAsync netcoreapp3.1 1.63μs 0.899ns 3.36ns 0.0138 0 0 1.02 KB
master CallElasticsearchAsync net472 2.5μs 1.67ns 6.49ns 0.167 0 0 1.05 KB
#6080 CallElasticsearch net6.0 1.21μs 0.655ns 2.45ns 0.0133 0 0 976 B
#6080 CallElasticsearch netcoreapp3.1 1.49μs 1.29ns 4.82ns 0.0127 0 0 976 B
#6080 CallElasticsearch net472 2.49μs 2ns 7.74ns 0.158 0 0 995 B
#6080 CallElasticsearchAsync net6.0 1.11μs 0.419ns 1.57ns 0.0133 0 0 952 B
#6080 CallElasticsearchAsync netcoreapp3.1 1.63μs 0.659ns 2.46ns 0.0138 0 0 1.02 KB
#6080 CallElasticsearchAsync net472 2.7μs 2.45ns 9.5ns 0.166 0 0 1.05 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.25μs 0.791ns 2.96ns 0.0131 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.66μs 0.784ns 2.93ns 0.0126 0 0 952 B
master ExecuteAsync net472 1.85μs 0.991ns 3.84ns 0.145 0 0 915 B
#6080 ExecuteAsync net6.0 1.3μs 0.859ns 3.21ns 0.0135 0 0 952 B
#6080 ExecuteAsync netcoreapp3.1 1.64μs 0.458ns 1.65ns 0.013 0 0 952 B
#6080 ExecuteAsync net472 1.75μs 1.02ns 3.94ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.27μs 1.72ns 6.19ns 0.0298 0 0 2.22 KB
master SendAsync netcoreapp3.1 5.02μs 2.03ns 7.58ns 0.0376 0 0 2.76 KB
master SendAsync net472 7.76μs 4.41ns 17.1ns 0.497 0 0 3.15 KB
#6080 SendAsync net6.0 4.23μs 2.15ns 8.31ns 0.0318 0 0 2.22 KB
#6080 SendAsync netcoreapp3.1 5.24μs 2.11ns 8.16ns 0.0368 0 0 2.76 KB
#6080 SendAsync net472 7.8μs 26.8ns 104ns 0.496 0 0 3.15 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.58μs 0.646ns 2.5ns 0.0229 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 2.12μs 0.738ns 2.66ns 0.0224 0 0 1.64 KB
master EnrichedLog net472 2.53μs 0.74ns 2.87ns 0.249 0 0 1.57 KB
#6080 EnrichedLog net6.0 1.43μs 0.674ns 2.52ns 0.0232 0 0 1.64 KB
#6080 EnrichedLog netcoreapp3.1 2.33μs 1.12ns 4.2ns 0.0222 0 0 1.64 KB
#6080 EnrichedLog net472 2.69μs 1.12ns 4.35ns 0.249 0 0 1.57 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 113μs 130ns 502ns 0.0566 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 118μs 189ns 732ns 0.0586 0 0 4.28 KB
master EnrichedLog net472 146μs 190ns 735ns 0.655 0.218 0 4.46 KB
#6080 EnrichedLog net6.0 115μs 134ns 500ns 0.0572 0 0 4.28 KB
#6080 EnrichedLog netcoreapp3.1 118μs 257ns 962ns 0 0 0 4.28 KB
#6080 EnrichedLog net472 146μs 124ns 482ns 0.658 0.219 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.15μs 1.18ns 4.58ns 0.0315 0 0 2.2 KB
master EnrichedLog netcoreapp3.1 4.22μs 1.84ns 7.12ns 0.0297 0 0 2.2 KB
master EnrichedLog net472 4.79μs 0.811ns 3.03ns 0.319 0 0 2.02 KB
#6080 EnrichedLog net6.0 3.04μs 1.3ns 4.69ns 0.0316 0 0 2.2 KB
#6080 EnrichedLog netcoreapp3.1 4.19μs 2.27ns 8.8ns 0.0293 0 0 2.2 KB
#6080 EnrichedLog net472 4.95μs 1.01ns 3.91ns 0.32 0 0 2.02 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.28μs 0.547ns 2.05ns 0.0159 0 0 1.14 KB
master SendReceive netcoreapp3.1 1.76μs 2.08ns 8.04ns 0.0149 0 0 1.14 KB
master SendReceive net472 2.07μs 1.1ns 4.26ns 0.183 0 0 1.16 KB
#6080 SendReceive net6.0 1.29μs 2.2ns 8.23ns 0.0162 0 0 1.14 KB
#6080 SendReceive netcoreapp3.1 1.85μs 1.84ns 7.12ns 0.0148 0 0 1.14 KB
#6080 SendReceive net472 2.05μs 1.56ns 5.61ns 0.184 0.00105 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.72μs 0.993ns 3.84ns 0.0217 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 3.99μs 0.803ns 3.11ns 0.022 0 0 1.65 KB
master EnrichedLog net472 4.35μs 2.4ns 9.31ns 0.323 0 0 2.04 KB
#6080 EnrichedLog net6.0 2.86μs 0.961ns 3.6ns 0.0215 0 0 1.6 KB
#6080 EnrichedLog netcoreapp3.1 4.02μs 0.936ns 3.5ns 0.0221 0 0 1.65 KB
#6080 EnrichedLog net472 4.47μs 2.6ns 9.74ns 0.322 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 398ns 0.438ns 1.7ns 0.00817 0 0 576 B
master StartFinishSpan netcoreapp3.1 567ns 1.18ns 4.58ns 0.00776 0 0 576 B
master StartFinishSpan net472 617ns 0.362ns 1.4ns 0.0917 0 0 578 B
master StartFinishScope net6.0 605ns 0.514ns 1.85ns 0.00967 0 0 696 B
master StartFinishScope netcoreapp3.1 656ns 0.653ns 2.53ns 0.00952 0 0 696 B
master StartFinishScope net472 796ns 0.99ns 3.83ns 0.104 0 0 658 B
#6080 StartFinishSpan net6.0 410ns 0.304ns 1.18ns 0.008 0 0 576 B
#6080 StartFinishSpan netcoreapp3.1 553ns 0.639ns 2.39ns 0.00768 0 0 576 B
#6080 StartFinishSpan net472 669ns 1.03ns 4ns 0.0917 0 0 578 B
#6080 StartFinishScope net6.0 586ns 0.29ns 1.09ns 0.00967 0 0 696 B
#6080 StartFinishScope netcoreapp3.1 726ns 0.47ns 1.82ns 0.00945 0 0 696 B
#6080 StartFinishScope net472 833ns 0.281ns 1.01ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 609ns 0.389ns 1.51ns 0.00974 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 903ns 0.481ns 1.86ns 0.00951 0 0 696 B
master RunOnMethodBegin net472 1.13μs 0.878ns 3.4ns 0.104 0 0 658 B
#6080 RunOnMethodBegin net6.0 644ns 0.326ns 1.26ns 0.00974 0 0 696 B
#6080 RunOnMethodBegin netcoreapp3.1 887ns 1.29ns 4.65ns 0.00921 0 0 696 B
#6080 RunOnMethodBegin net472 1.13μs 0.466ns 1.8ns 0.105 0 0 658 B

@tonyredondo tonyredondo merged commit b966684 into master Sep 26, 2024
62 of 66 checks passed
@tonyredondo tonyredondo deleted the tony/civisibility-add-support-mstest-3_6_0 branch September 26, 2024 16:52
@github-actions github-actions bot added this to the vNext-v3 milestone Sep 26, 2024
@andrewlock andrewlock added the type:enhancement Improvement to an existing feature label Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewlock andrewlock andrewlock approved these changes

Assignees

@tonyredondo tonyredondo

Labels
area:ci-visibility type:enhancement Improvement to an existing feature
Projects
None yet
Milestone
3.4.0
Development

Successfully merging this pull request may close these issues.
2 participants
@tonyredondo @andrewlock