Skip to content

Create a simple tool for comparing results of different benchmark runs #161

@adamsitnik

Description

@adamsitnik

From @AndyAyersMS

What would be super-useful is an overall rollup of regressions / improvements sorted by relative magnitude. With 300 things to look at we need to prioritize somehow.
It’s also been interesting in the past to compare across OSes – eg we know Linux is a bit slower than Windows in certain areas, but by and large we hope to have rough parity.
And likewise across arches, we would again hope for fairly consistent ratios for x86/x64, arm64/x64, etc.

My idea: write plain simple tool that import json files from two provided folders, does the statistical test and prints summary ordered descending by magnitude

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions