Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make it easier to compare different benchmarks #151

Open
Porges opened this issue Jul 20, 2017 · 4 comments
Open

Make it easier to compare different benchmarks #151

Porges opened this issue Jul 20, 2017 · 4 comments

Comments

@Porges
Copy link

Porges commented Jul 20, 2017

When comparing multiple different benchmarks, it would be nice to have a way to mark them as "comparable" (perhaps bcomparisongroup), with the following effects:

  • numbers are all generated on the same time scale (e.g. all 'μs' or all 'ms', not a mixture)
  • graphs are produced with the same vertical scale

This would make it easier to eyeball differences.

As an extension, perhaps also the library could do some statistics to see if there are true differences between the timings (ANOVA? it's been a while...), but that would be an added bonus.

@Porges
Copy link
Author

Porges commented Jul 20, 2017

Also somewhat related to #62.

@cricketer94
Copy link

cricketer94 commented Aug 16, 2019

Just wonder in relation to this issue, whether there is any way to "scale" the PDF values given by criterion report?

Context: I am running a lot of individual benchmarks and would like to compare them on a KDE graph I am trying to render using the KDE information in an exported JSON report. However, since the y-value scales are inconsistent it is hard to compare different benchmarks. If there was a way to even make these scales even roughly comparable, that would be extremely helpful.

EDIT
It would be useful to for example pass in a range [0,1] such that alll the kdePDf values are rescaled to fall within this value range

@NorfairKing
Copy link

numbers are all generated on the same time scale (e.g. all 'μs' or all 'ms', not a mixture)

Yes please

@RyanGlScott
Copy link
Member

I was recently made aware of the criterion-compare tool. I'd be willing to incorporate similar functionality in criterion, should someone want to prepare a patch for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants