Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CI check to verify that benchmark queries return the expected results #4015

Merged
merged 9 commits into from
Oct 30, 2022

Conversation

andygrove
Copy link
Member

@andygrove andygrove commented Oct 29, 2022

Which issue does this PR close?

Closes #4016

Rationale for this change

I would like to test that the benchmarks are producing the correct results so that we catch regressions. Some of the queries are producing very wrong results.

What changes are included in this PR?

  • Add ci feature to benchmark crate, which will verify results are correct
  • Ignore failing tests
  • Add new CI workflow
  • Make benchmark crate usable as a dependency so that Ballista can use it (and avoid maintaining two copies of this code)

Output:

verify

running 22 tests
test tests::verify_q1 ... ok
test tests::verify_q10 ... ok
test tests::verify_q11 ... ignored
test tests::verify_q12 ... ok
test tests::verify_q13 ... ok
test tests::verify_q14 ... ignored
test tests::verify_q15 ... ok
test tests::verify_q16 ... ok
test tests::verify_q17 ... ignored
test tests::verify_q18 ... ok
test tests::verify_q19 ... ok
test tests::verify_q2 ... ok
test tests::verify_q20 ... ok
test tests::verify_q21 ... ok
test tests::verify_q22 ... ok
test tests::verify_q3 ... ok
test tests::verify_q4 ... ok
test tests::verify_q5 ... ok
test tests::verify_q6 ... ignored
test tests::verify_q7 ... ok
test tests::verify_q8 ... ok
test tests::verify_q9 ... ignored

Are there any user-facing changes?

@andygrove andygrove marked this pull request as draft October 29, 2022 13:38
@andygrove andygrove marked this pull request as ready for review October 29, 2022 14:42
@andygrove andygrove marked this pull request as draft October 29, 2022 15:00
@andygrove andygrove changed the title Test benchmarks in CI Add CI check to verify that benchmark queries return the expected results Oct 29, 2022
@andygrove andygrove marked this pull request as ready for review October 29, 2022 18:27

# the release profile takes a long time to build so we can use this profile during development to save time
# cargo build --profile release-nonlto
[profile.release-nonlto]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CI workflow uses this profile rather than release to try and save some time

Copy link
Contributor

@Dandandan Dandandan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@andygrove andygrove merged commit 6651070 into apache:master Oct 30, 2022
@andygrove andygrove deleted the ci-test-benchmarks branch October 30, 2022 14:17
@ursabot
Copy link

ursabot commented Oct 30, 2022

Benchmark runs are scheduled for baseline = d391b85 and contender = 6651070. 6651070 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

jimexist pushed a commit to jimexist/arrow-datafusion that referenced this pull request Oct 31, 2022
Dandandan pushed a commit to yuuch/arrow-datafusion that referenced this pull request Nov 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Publish benchmark crate so that it can be used as a library in Ballista
3 participants