From @AndyAyersMS
What would be super-useful is an overall rollup of regressions / improvements sorted by relative magnitude. With 300 things to look at we need to prioritize somehow.
It’s also been interesting in the past to compare across OSes – eg we know Linux is a bit slower than Windows in certain areas, but by and large we hope to have rough parity.
And likewise across arches, we would again hope for fairly consistent ratios for x86/x64, arm64/x64, etc.
My idea: write plain simple tool that import json files from two provided folders, does the statistical test and prints summary ordered descending by magnitude