On-demand generation of comparison report for current and previous release of dotnet/runtime

[matouskozak]

Description

To improve microbenchmark triaging efficiency, it would be beneficial to have a report (e.g., in a form of dashboard) comparing the previous .NET release results with the current dotnet/runtime main. E.g., comparing dotnet/runtime:release/9.0 and dotnet/runtime:main measurements.

The report should list all new regressions/improvements based on comparison between the two sets of measurements. To reduce false positives caused by bimodal and noisy microbenchmark, an averaging of the last X measurements could be introduced to smooth the graphs.

Additionally, the report should allow to filter-out microbenchmarks which improved/regressed by less than X %.

Implementation Idea

1. Filter related microbenchmark results
   • filter out all measurements where BuildBranch=ref/heads/main
   • filter out all measurements where BuildBranch=ref/heads/release/9.0
2. Average the results from past X days based on Configuration (e.g., Mono x64 AOT LLVM) and TestName (microbenchmark name).
3. For each Configuration and TestName, calculate the diff.
4. Filter-out measurements with diff less than specified threshold.
5. Present the report
   • Individual regressions/improvements (Configuration + TestName) sorted by magnitude.
   • Number of regressions/improvements per Configuration

---

cc: @kotlarmilos