We use two test suites for benchmarking: clickbench against the standard 100M row
(80GB) dataset, and a data warehouse benchmark suite similar to TPC-H, with a
10GB dataset. These suites are called clickbench-100M and warehouse-10G
respectively. The benchmarking code is open source.
The high level results we track are below: the total query execution time of the
SELECT queries run in the benchmark (i.e. without setup or data load time)
of each suite. Every query is individually tracked in Bencher,
and can be explored in detail on our Bencher project page.
All benchmarks are query time, reported in milliseconds. Lower values are better.
Hydra benchmarks each of its releases. These tests are run on a c6a.4xlarge
instance (16 vCPU, 32 GiB RAM) with 500 GiB of GP2 storage.
Hydra uses Bencher to continuously track benchmarks after every
commit to main. This allows us to determine the impact of performance improvements
as well as track performance regressions.
These tests use the open source Docker image and run on GitHub's 4-core test runners. You can review the GitHub action for more methodology details.
Since many commits will not affect performance, it is normal for graphs to be principally flat for many commits in a row. These graphs do not have historical data from previous releases.