Shell script to collect benchmarks for multiple versions

[logan-keede]

Which issue does this PR close?

• Part of Run DataFusion benchmarks regularly and track performance history over time #5504

Rationale for this change

Here is a suggestion on how to proceed with this project:

1. Create the converter from bench json --> line protocol (e.g. Export benchmark information as line protocol #6107)
2. Write a script that runs the bench.sh script to gather the clickbench performance numbers over the last 5 releases:

git checkout 40.0.0
./bench.sh run clickbench_partitioned
git checkout 41.0.0
./bench.sh run clickbench_partitioned
...
git checkout 45.0.0
./bench.sh run clickbench_partitioned

And then load/plot that data using Grafana.

Originally posted by @alamb in #5504

What changes are included in this PR?

Just a simple script to collect benchmarks for last 5 releases

Are these changes tested?

Yes, by running the script on my laptop.
using sh collect_bench.sh tpch.

Are there any user-facing changes?

Nope.

[logan-keede]

One problem that I sometimes encounter is that cargo decides to use arrow-arith v53.4.0 for particular releases which ends up giving compilation error.
I’m not sure why this happens. Sometimes, for the same script and same release, cargo decides to use some other version of arrow-arith and everything works fine.
If anyone knows a workaround, please let me know.

[alamb]

Sorry @logan-keede -- I am very excited to try this one out, but I ran otu of time today

[logan-keede]

Sorry @logan-keede -- I am very excited to try this one out, but I ran otu of time today

No problem at all! I look forward to your feedback whenever you have the time.

[comphead]

Thanks @logan-keede that is pretty neat, appreciate if you document the script and its usage in benchmarks/README.md

[logan-keede]

Just noticed that benchmarks/README.md is not included in prettier CI check, is that intended?

[alamb]

Just noticed that benchmarks/README.md is not included in prettier CI check, is that intended?

I don't think it is intended

And I am sorry I haven't had time to test out this PR yet

[logan-keede]

Just noticed that benchmarks/README.md is not included in prettier CI check, is that intended?

I don't think it is intended

And I am sorry I haven't had time to test out this PR yet

I pushed a commit to add benchmarks to prettier check list.

No problem, I understand it might be a bit to lengthy and resource intensive to test.
If so, you could try limiting it to 4 threads and edit cargo command in bench.sh to remove release flag(temp) while handling other tasks(may not work with other builds though). Otherwise, feel free to ignore this.

[alamb]

Thank you for this work @logan-keede -- I am sorry for the very long delay. It seems @saraghds is also working on this script so I am going to try and help it along.

I merged this branch up from main and added some more comments. I also would like to make the versions it checks a bit more configurable -- I will test and work on that over the day

[alamb]

I want this script to be useable by more people so I think it is important to document it a bit more

I am also going to investigate potentially overriding the list of git commits to run rather than assuming what versions.

[alamb]

Sorry for the long delay in feedback -- I have been working on other things.

What I really want is a way to track DataFusion performance over time during development (so not released version numbers) so that as new changes are added to the code we can see how the overall trend is doing.

Given the initial state of this PR, the best they can do is run once a month after a release is done.

So since they don't really satisfy the need yet (being able to get a handle on ongoing performane) I haven't spent a lot of time reviewing / working them.

BTW here is the kind of thing I want to see / create:
https://benchmarks.mikemccandless.com/