Skip to content

Feature: Add lance compress benchmark #4907

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
connortsui20 merged 2 commits into from
Oct 10, 2025
Merged

Feature: Add lance compress benchmark #4907

connortsui20 merged 2 commits into from
Oct 10, 2025

Conversation

@connortsui20
Copy link
Contributor

@connortsui20 connortsui20 commented Oct 9, 2025

Additionally cleans up some other things.

@connortsui20 connortsui20 added changelog/chore A trivial change changelog/feature A new feature and removed changelog/chore A trivial change labels Oct 9, 2025
@codecov
Copy link

codecov bot commented Oct 9, 2025 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.52%. Comparing base (23cefb0) to head (aa9d87a).
⚠️ Report is 4 commits behind head on develop.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@connortsui20 connortsui20 added the action/benchmark Trigger full benchmarks to run on this PR label Oct 9, 2025
joseph-isaacs
joseph-isaacs reviewed Oct 9, 2025
View reviewed changes
bench-vortex/src/bin/compress.rs
value_delimiter = ',',
value_enum,
default_values_t = vec![Format::Parquet, Format::OnDiskVortex]
default_values_t = vec![Format::Parquet, Format::Lance, Format::OnDiskVortex]
Copy link
Contributor

@joseph-isaacs joseph-isaacs Oct 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no default please?

Copy link
Contributor

@joseph-isaacs joseph-isaacs Oct 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we add a feature flag TODO?

@connortsui20 connortsui20 added action/benchmark Trigger full benchmarks to run on this PR changelog/feature A new feature and removed changelog/feature A new feature action/benchmark Trigger full benchmarks to run on this PR labels Oct 9, 2025
@connortsui20 connortsui20 removed the changelog/feature A new feature label Oct 9, 2025
@connortsui20 connortsui20 changed the base branch from ct/refactor-compress-bench to develop October 9, 2025 20:13
joseph-isaacs
joseph-isaacs reviewed Oct 9, 2025
View reviewed changes
@connortsui20 connortsui20 added changelog/feature A new feature action/benchmark Trigger full benchmarks to run on this PR and removed action/benchmark Trigger full benchmarks to run on this PR labels Oct 9, 2025
@github-actions github-actions bot removed the action/benchmark Trigger full benchmarks to run on this PR label Oct 9, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Oct 9, 2025 •
edited
Loading

Benchmarks: Random Access

Summary

  • Overall: 0.993x ➖
  • Vortex: 0.984x ➖
  • Best: random-access/vortex-tokio-local-disk (0.984x)
  • Worst: No regressions
  • Significant (>10%): 0↑ 0↓
Detailed Results Table
name PR daa3bf0 base d711afb ratio (PR/base) unit remark
random-access/parquet-tokio-local-disk 182024578 1.80304e+08 1.00954 ns
random-access/lance-tokio-local-disk 968869 983117 0.985507 ns
random-access/vortex-tokio-local-disk 1263108 1.28414e+06 0.983621 ns

joseph-isaacs
joseph-isaacs reviewed Oct 9, 2025
View reviewed changes
@github-actions
Copy link
Contributor

github-actions bot commented Oct 9, 2025 •
edited
Loading

Benchmarks: FineWeb

Summary

  • Overall: 1.011x ➖
  • Vortex: 1.009x ➖
  • duckdb:vortex: 1.011x ➖
  • datafusion:vortex: 1.007x ➖
  • Best: fineweb_q01/DataFusion:vortex-file-compressed (0.914x)
  • Worst: fineweb_q00/DataFusion:vortex-compact (1.096x)
  • Significant (>10%): 0↑ 0↓
Detailed Results Table
name PR daa3bf0 base d711afb ratio (PR/base) unit remark
fineweb_q00/DuckDB:parquet 29479351 2.78211e+07 1.0596 ns
fineweb_q01/DuckDB:parquet 97883610 9.54384e+07 1.02562 ns
fineweb_q02/DuckDB:parquet 102755024 9.8747e+07 1.04059 ns
fineweb_q03/DuckDB:parquet 305454006 3.02219e+08 1.0107 ns
fineweb_q04/DuckDB:parquet 432985803 4.25345e+08 1.01796 ns
fineweb_q05/DuckDB:parquet 406295502 4.01171e+08 1.01277 ns
fineweb_q06/DuckDB:parquet 204045129 1.9971e+08 1.02171 ns
fineweb_q07/DuckDB:parquet 214849383 2.12798e+08 1.00964 ns
fineweb_q08/DuckDB:parquet 60998281 5.86685e+07 1.03971 ns
fineweb_q00/DuckDB:vortex-file-compressed 4129624 4.16389e+06 0.99177 ns
fineweb_q01/DuckDB:vortex-file-compressed 23704356 2.5524e+07 0.928708 ns
fineweb_q02/DuckDB:vortex-file-compressed 29355444 2.99859e+07 0.978975 ns
fineweb_q03/DuckDB:vortex-file-compressed 114462335 1.12172e+08 1.02042 ns
fineweb_q04/DuckDB:vortex-file-compressed 188732535 1.8837e+08 1.00192 ns
fineweb_q05/DuckDB:vortex-file-compressed 170003997 1.69457e+08 1.00323 ns
fineweb_q06/DuckDB:vortex-file-compressed 57625919 5.26709e+07 1.09408 ns
fineweb_q07/DuckDB:vortex-file-compressed 56835972 5.67748e+07 1.00108 ns
fineweb_q08/DuckDB:vortex-file-compressed 7532232 7.61483e+06 0.989153 ns
fineweb_q00/DuckDB:vortex-compact 5502629 5.14743e+06 1.069 ns
fineweb_q01/DuckDB:vortex-compact 88477112 8.58847e+07 1.03018 ns
fineweb_q02/DuckDB:vortex-compact 94850497 9.36348e+07 1.01298 ns
fineweb_q03/DuckDB:vortex-compact 844150293 8.33687e+08 1.01255 ns
fineweb_q04/DuckDB:vortex-compact 887036800 8.85939e+08 1.00124 ns
fineweb_q05/DuckDB:vortex-compact 792948948 7.88668e+08 1.00543 ns
fineweb_q06/DuckDB:vortex-compact 438396862 4.31597e+08 1.01576 ns
fineweb_q07/DuckDB:vortex-compact 462210698 4.49223e+08 1.02891 ns
fineweb_q08/DuckDB:vortex-compact 7262235 7.11904e+06 1.02011 ns
fineweb_q00/DataFusion:parquet 6170230 5.91453e+06 1.04323 ns
fineweb_q01/DataFusion:parquet 297913482 2.97073e+08 1.00283 ns
fineweb_q02/DataFusion:parquet 299336242 3.00238e+08 0.996997 ns
fineweb_q03/DataFusion:parquet 296619393 2.97541e+08 0.996903 ns
fineweb_q04/DataFusion:parquet 319078210 3.17107e+08 1.00622 ns
fineweb_q05/DataFusion:parquet 309521052 3.10158e+08 0.997945 ns
fineweb_q06/DataFusion:parquet 301505769 3.00617e+08 1.00296 ns
fineweb_q07/DataFusion:parquet 297531812 2.97602e+08 0.999764 ns
fineweb_q08/DataFusion:parquet 293862084 2.93123e+08 1.00252 ns
fineweb_q00/DataFusion:vortex-file-compressed 3246192 3.40181e+06 0.954254 ns
fineweb_q01/DataFusion:vortex-file-compressed 21648018 2.36787e+07 0.914242 ns
fineweb_q02/DataFusion:vortex-file-compressed 22034768 2.35612e+07 0.935214 ns
fineweb_q03/DataFusion:vortex-file-compressed 71134316 6.65538e+07 1.06882 ns
fineweb_q04/DataFusion:vortex-file-compressed 196462837 2.00309e+08 0.980799 ns
fineweb_q05/DataFusion:vortex-file-compressed 176814048 1.82796e+08 0.967276 ns
fineweb_q06/DataFusion:vortex-file-compressed 65261890 6.21109e+07 1.05073 ns
fineweb_q07/DataFusion:vortex-file-compressed 66235512 6.5332e+07 1.01383 ns
fineweb_q08/DataFusion:vortex-file-compressed 7981160 7.45162e+06 1.07106 ns
fineweb_q00/DataFusion:vortex-compact 3905450 3.56296e+06 1.09612 ns
fineweb_q01/DataFusion:vortex-compact 104470174 1.04981e+08 0.995135 ns
fineweb_q02/DataFusion:vortex-compact 102332549 9.42552e+07 1.0857 ns
fineweb_q03/DataFusion:vortex-compact 939532964 9.45455e+08 0.993736 ns
fineweb_q04/DataFusion:vortex-compact 1060490745 1.05532e+09 1.0049 ns
fineweb_q05/DataFusion:vortex-compact 898014879 8.85185e+08 1.01449 ns
fineweb_q06/DataFusion:vortex-compact 484653722 4.79749e+08 1.01022 ns
fineweb_q07/DataFusion:vortex-compact 497637936 5.00269e+08 0.994741 ns
fineweb_q08/DataFusion:vortex-compact 6809980 6.77612e+06 1.005 ns

@github-actions
Copy link
Contributor

github-actions bot commented Oct 9, 2025 •
edited
Loading

Benchmarks: TPC-H SF=1 on NVME

Summary

  • Overall: 1.023x ➖
  • Vortex: 1.024x ➖
  • duckdb:vortex: 1.030x ➖
  • datafusion:vortex: 1.018x ➖
  • Best: tpch_q10/DataFusion:vortex-file-compressed (0.939x)
  • Worst: tpch_q07/DuckDB:vortex-file-compressed (1.125x)
  • Significant (>10%): 0↑ 2↓
Detailed Results Table
name PR daa3bf0 base d711afb ratio (PR/base) unit remark
tpch_q01/DataFusion:arrow 59881670 6.11837e+07 0.97872 ns
tpch_q02/DataFusion:arrow 18035093 1.74489e+07 1.0336 ns
tpch_q03/DataFusion:arrow 22680937 2.21924e+07 1.02201 ns
tpch_q04/DataFusion:arrow 12441896 1.21639e+07 1.02285 ns
tpch_q05/DataFusion:arrow 32483443 3.29945e+07 0.984511 ns
tpch_q06/DataFusion:arrow 7268883 8.78479e+06 0.82744 ns
tpch_q07/DataFusion:arrow 51674748 5.18816e+07 0.996014 ns
tpch_q08/DataFusion:arrow 18591311 1.87478e+07 0.991654 ns
tpch_q09/DataFusion:arrow 31854166 3.16674e+07 1.0059 ns
tpch_q10/DataFusion:arrow 27364274 2.76178e+07 0.990819 ns
tpch_q11/DataFusion:arrow 39107928 3.82659e+07 1.022 ns
tpch_q12/DataFusion:arrow 19092412 1.88184e+07 1.01456 ns
tpch_q13/DataFusion:arrow 17952749 1.75267e+07 1.02431 ns
tpch_q14/DataFusion:arrow 7474962 7.21709e+06 1.03573 ns
tpch_q15/DataFusion:arrow 13698365 1.28745e+07 1.06399 ns
tpch_q16/DataFusion:arrow 15103610 1.44e+07 1.04886 ns
tpch_q17/DataFusion:arrow 51376270 4.85937e+07 1.05726 ns
tpch_q18/DataFusion:arrow 104227853 9.62523e+07 1.08286 ns
tpch_q19/DataFusion:arrow 16905319 1.64111e+07 1.03011 ns
tpch_q20/DataFusion:arrow 21014994 1.98609e+07 1.05811 ns
tpch_q21/DataFusion:arrow 75554558 7.51129e+07 1.00588 ns
tpch_q22/DataFusion:arrow 10507190 1.00124e+07 1.04942 ns
tpch_q01/DataFusion:parquet 154388382 1.52143e+08 1.01476 ns
tpch_q02/DataFusion:parquet 95283562 8.51657e+07 1.1188 ns
tpch_q03/DataFusion:parquet 81817034 8.04079e+07 1.01752 ns
tpch_q04/DataFusion:parquet 56461131 5.26426e+07 1.07254 ns
tpch_q05/DataFusion:parquet 101057828 1.00168e+08 1.00889 ns
tpch_q06/DataFusion:parquet 34259740 3.32248e+07 1.03115 ns
tpch_q07/DataFusion:parquet 129857918 1.30977e+08 0.991458 ns
tpch_q08/DataFusion:parquet 124804717 1.23894e+08 1.00735 ns
tpch_q09/DataFusion:parquet 145680937 1.48103e+08 0.983648 ns
tpch_q10/DataFusion:parquet 123536086 1.17889e+08 1.04791 ns
tpch_q11/DataFusion:parquet 67272613 6.50771e+07 1.03374 ns
tpch_q12/DataFusion:parquet 78460842 7.87242e+07 0.996655 ns
tpch_q13/DataFusion:parquet 198757593 1.96499e+08 1.0115 ns
tpch_q14/DataFusion:parquet 44733802 4.17637e+07 1.07112 ns
tpch_q15/DataFusion:parquet 54834284 5.58501e+07 0.981812 ns
tpch_q16/DataFusion:parquet 46362820 4.64589e+07 0.997932 ns
tpch_q17/DataFusion:parquet 144615337 1.26976e+08 1.13891 ns
tpch_q18/DataFusion:parquet 181991249 1.82638e+08 0.996459 ns
tpch_q19/DataFusion:parquet 87245982 8.95794e+07 0.973951 ns
tpch_q20/DataFusion:parquet 71786540 6.6838e+07 1.07404 ns
tpch_q21/DataFusion:parquet 143285882 1.3035e+08 1.09924 ns
tpch_q22/DataFusion:parquet 67981312 6.51276e+07 1.04382 ns
tpch_q01/DataFusion:vortex-file-compressed 57183654 5.52399e+07 1.03519 ns
tpch_q02/DataFusion:vortex-file-compressed 34390178 3.44509e+07 0.998236 ns
tpch_q03/DataFusion:vortex-file-compressed 22905449 2.24583e+07 1.01991 ns
tpch_q04/DataFusion:vortex-file-compressed 16400474 1.56866e+07 1.04551 ns
tpch_q05/DataFusion:vortex-file-compressed 59820260 5.87702e+07 1.01787 ns
tpch_q06/DataFusion:vortex-file-compressed 7430247 7.35746e+06 1.00989 ns
tpch_q07/DataFusion:vortex-file-compressed 71714896 7.04725e+07 1.01763 ns
tpch_q08/DataFusion:vortex-file-compressed 45184237 4.22449e+07 1.06958 ns
tpch_q09/DataFusion:vortex-file-compressed 58898029 5.59568e+07 1.05256 ns
tpch_q10/DataFusion:vortex-file-compressed 37113056 3.9545e+07 0.938502 ns
tpch_q11/DataFusion:vortex-file-compressed 21773689 2.20391e+07 0.987959 ns
tpch_q12/DataFusion:vortex-file-compressed 17770892 1.73133e+07 1.02643 ns
tpch_q13/DataFusion:vortex-file-compressed 24934908 2.49218e+07 1.00053 ns
tpch_q14/DataFusion:vortex-file-compressed 10994418 1.07959e+07 1.01839 ns
tpch_q15/DataFusion:vortex-file-compressed 18635928 1.847e+07 1.00898 ns
tpch_q16/DataFusion:vortex-file-compressed 21534933 2.09167e+07 1.02956 ns
tpch_q17/DataFusion:vortex-file-compressed 59045795 5.81315e+07 1.01573 ns
tpch_q18/DataFusion:vortex-file-compressed 91143126 9.06081e+07 1.0059 ns
tpch_q19/DataFusion:vortex-file-compressed 18195400 1.778e+07 1.02336 ns
tpch_q20/DataFusion:vortex-file-compressed 25142455 2.39028e+07 1.05186 ns
tpch_q21/DataFusion:vortex-file-compressed 72484615 7.08676e+07 1.02282 ns
tpch_q22/DataFusion:vortex-file-compressed 13341859 1.34902e+07 0.989001 ns
tpch_q01/DataFusion:vortex-compact 61129245 5.53184e+07 1.10504 ns
tpch_q02/DataFusion:vortex-compact 39085056 4.0349e+07 0.968674 ns
tpch_q03/DataFusion:vortex-compact 25434245 2.60853e+07 0.97504 ns
tpch_q04/DataFusion:vortex-compact 18808688 1.88485e+07 0.997888 ns
tpch_q05/DataFusion:vortex-compact 61972239 6.26452e+07 0.989257 ns
tpch_q06/DataFusion:vortex-compact 11673416 1.19618e+07 0.975893 ns
tpch_q07/DataFusion:vortex-compact 76969796 7.31364e+07 1.05241 ns
tpch_q08/DataFusion:vortex-compact 43666823 4.33428e+07 1.00748 ns
tpch_q09/DataFusion:vortex-compact 61865601 6.21077e+07 0.996102 ns
tpch_q10/DataFusion:vortex-compact 44451623 4.35321e+07 1.02112 ns
tpch_q11/DataFusion:vortex-compact 23770246 2.25276e+07 1.05516 ns
tpch_q12/DataFusion:vortex-compact 26751653 2.63387e+07 1.01568 ns
tpch_q13/DataFusion:vortex-compact 31838518 3.01844e+07 1.0548 ns
tpch_q14/DataFusion:vortex-compact 16441058 1.63911e+07 1.00305 ns
tpch_q15/DataFusion:vortex-compact 29297574 2.83443e+07 1.03363 ns
tpch_q16/DataFusion:vortex-compact 26891896 2.6519e+07 1.01406 ns
tpch_q17/DataFusion:vortex-compact 63399940 6.34966e+07 0.998478 ns
tpch_q18/DataFusion:vortex-compact 93189059 9.16256e+07 1.01706 ns
tpch_q19/DataFusion:vortex-compact 34881023 3.40218e+07 1.02526 ns
tpch_q20/DataFusion:vortex-compact 29855997 2.93116e+07 1.01857 ns
tpch_q21/DataFusion:vortex-compact 80443176 7.63017e+07 1.05428 ns
tpch_q22/DataFusion:vortex-compact 16014808 1.52949e+07 1.04707 ns
tpch_q01/DuckDB:parquet 80197069 7.98172e+07 1.00476 ns
tpch_q02/DuckDB:parquet 48253181 4.52506e+07 1.06635 ns
tpch_q03/DuckDB:parquet 78549591 7.66251e+07 1.02512 ns
tpch_q04/DuckDB:parquet 60114303 5.99134e+07 1.00335 ns
tpch_q05/DuckDB:parquet 75789420 7.43719e+07 1.01906 ns
tpch_q06/DuckDB:parquet 22535411 2.2863e+07 0.985673 ns
tpch_q07/DuckDB:parquet 81791024 8.02671e+07 1.01899 ns
tpch_q08/DuckDB:parquet 95942770 9.59371e+07 1.00006 ns
tpch_q09/DuckDB:parquet 149421064 1.442e+08 1.03621 ns
tpch_q10/DuckDB:parquet 149629376 1.37443e+08 1.08866 ns
tpch_q11/DuckDB:parquet 28235071 2.55078e+07 1.10692 ns
tpch_q12/DuckDB:parquet 61156099 5.8896e+07 1.03837 ns
tpch_q13/DuckDB:parquet 289146801 2.70936e+08 1.06721 ns
tpch_q14/DuckDB:parquet 53425904 5.19958e+07 1.0275 ns
tpch_q15/DuckDB:parquet 29626800 2.94542e+07 1.00586 ns
tpch_q16/DuckDB:parquet 60423612 5.86547e+07 1.03016 ns
tpch_q17/DuckDB:parquet 59293649 5.86072e+07 1.01171 ns
tpch_q18/DuckDB:parquet 123951022 1.20856e+08 1.02561 ns
tpch_q19/DuckDB:parquet 73369553 7.0461e+07 1.04128 ns
tpch_q20/DuckDB:parquet 64875470 6.28285e+07 1.03258 ns
tpch_q21/DuckDB:parquet 181179239 1.87875e+08 0.964359 ns
tpch_q22/DuckDB:parquet 57059901 5.62191e+07 1.01496 ns
tpch_q01/DuckDB:vortex-file-compressed 28246438 2.8837e+07 0.979521 ns
tpch_q02/DuckDB:vortex-file-compressed 28957299 2.66031e+07 1.08849 ns
tpch_q03/DuckDB:vortex-file-compressed 31694179 2.91149e+07 1.08859 ns
tpch_q04/DuckDB:vortex-file-compressed 30888806 2.90399e+07 1.06367 ns
tpch_q05/DuckDB:vortex-file-compressed 33350802 3.28371e+07 1.01564 ns
tpch_q06/DuckDB:vortex-file-compressed 8766832 9.01406e+06 0.972573 ns
tpch_q07/DuckDB:vortex-file-compressed 40088625 3.5644e+07 1.12469 ns
tpch_q08/DuckDB:vortex-file-compressed 43427745 4.05097e+07 1.07203 ns
tpch_q09/DuckDB:vortex-file-compressed 68332454 6.86457e+07 0.995437 ns
tpch_q10/DuckDB:vortex-file-compressed 33033290 3.20095e+07 1.03198 ns
tpch_q11/DuckDB:vortex-file-compressed 16185596 1.53143e+07 1.05689 ns
tpch_q12/DuckDB:vortex-file-compressed 23881383 2.26278e+07 1.0554 ns
tpch_q13/DuckDB:vortex-file-compressed 38888696 3.65972e+07 1.06261 ns
tpch_q14/DuckDB:vortex-file-compressed 23560638 2.26467e+07 1.04036 ns
tpch_q15/DuckDB:vortex-file-compressed 12206739 1.2227e+07 0.998345 ns
tpch_q16/DuckDB:vortex-file-compressed 32795315 3.17124e+07 1.03415 ns
tpch_q17/DuckDB:vortex-file-compressed 24019892 2.40295e+07 0.999601 ns
tpch_q18/DuckDB:vortex-file-compressed 47797875 4.87972e+07 0.979521 ns
tpch_q19/DuckDB:vortex-file-compressed 32023985 3.10808e+07 1.03035 ns
tpch_q20/DuckDB:vortex-file-compressed 26218987 2.62426e+07 0.999102 ns
tpch_q21/DuckDB:vortex-file-compressed 100506935 9.99004e+07 1.00607 ns
tpch_q22/DuckDB:vortex-file-compressed 20071316 1.87307e+07 1.07157 ns
tpch_q01/DuckDB:vortex-compact 33105029 3.28349e+07 1.00823 ns
tpch_q02/DuckDB:vortex-compact 39613363 3.77732e+07 1.04872 ns
tpch_q03/DuckDB:vortex-compact 36789532 3.70479e+07 0.993027 ns
tpch_q04/DuckDB:vortex-compact 38584657 3.85225e+07 1.00161 ns
tpch_q05/DuckDB:vortex-compact 39711905 3.96071e+07 1.00265 ns
tpch_q06/DuckDB:vortex-compact 15525716 1.46989e+07 1.05625 ns
tpch_q07/DuckDB:vortex-compact 48679169 4.67963e+07 1.04023 ns
tpch_q08/DuckDB:vortex-compact 54682321 5.20304e+07 1.05097 ns
tpch_q09/DuckDB:vortex-compact 80336512 7.75706e+07 1.03566 ns
tpch_q10/DuckDB:vortex-compact 44346106 4.1781e+07 1.06139 ns
tpch_q11/DuckDB:vortex-compact 21506134 2.03959e+07 1.05444 ns
tpch_q12/DuckDB:vortex-compact 49194211 4.84819e+07 1.01469 ns
tpch_q13/DuckDB:vortex-compact 43372784 4.25619e+07 1.01905 ns
tpch_q14/DuckDB:vortex-compact 28832427 2.88618e+07 0.998982 ns
tpch_q15/DuckDB:vortex-compact 17850449 1.74686e+07 1.02186 ns
tpch_q16/DuckDB:vortex-compact 37844454 3.83157e+07 0.987701 ns
tpch_q17/DuckDB:vortex-compact 28794742 2.99946e+07 0.959999 ns
tpch_q18/DuckDB:vortex-compact 54175719 5.4439e+07 0.995164 ns
tpch_q19/DuckDB:vortex-compact 40568276 3.98442e+07 1.01817 ns
tpch_q20/DuckDB:vortex-compact 36741134 3.45243e+07 1.06421 ns
tpch_q21/DuckDB:vortex-compact 118416156 1.10458e+08 1.07205 ns
tpch_q22/DuckDB:vortex-compact 21929641 2.06553e+07 1.0617 ns
tpch_q01/DuckDB:duckdb 17316712 1.74247e+07 0.993804 ns
tpch_q02/DuckDB:duckdb 14031318 1.38552e+07 1.01271 ns
tpch_q03/DuckDB:duckdb 21016697 2.0835e+07 1.00872 ns
tpch_q04/DuckDB:duckdb 18970404 1.90318e+07 0.996773 ns
tpch_q05/DuckDB:duckdb 22087172 2.2514e+07 0.981042 ns
tpch_q06/DuckDB:duckdb 7184868 7.10422e+06 1.01135 ns
tpch_q07/DuckDB:duckdb 26337491 2.56188e+07 1.02805 ns
tpch_q08/DuckDB:duckdb 26341878 2.59623e+07 1.01462 ns
tpch_q09/DuckDB:duckdb 60384477 5.97789e+07 1.01013 ns
tpch_q10/DuckDB:duckdb 62235954 5.76458e+07 1.07963 ns
tpch_q11/DuckDB:duckdb 6925333 6.91389e+06 1.00165 ns
tpch_q12/DuckDB:duckdb 17361560 1.71951e+07 1.00968 ns
tpch_q13/DuckDB:duckdb 39425438 4.01044e+07 0.983069 ns
tpch_q14/DuckDB:duckdb 22952144 2.2115e+07 1.03785 ns
tpch_q15/DuckDB:duckdb 12138012 1.18134e+07 1.02748 ns
tpch_q16/DuckDB:duckdb 24980567 2.39125e+07 1.04466 ns
tpch_q17/DuckDB:duckdb 17593069 1.79636e+07 0.979374 ns
tpch_q18/DuckDB:duckdb 39214784 3.95263e+07 0.992118 ns
tpch_q19/DuckDB:duckdb 31712941 3.16193e+07 1.00296 ns
tpch_q20/DuckDB:duckdb 23187420 2.24228e+07 1.0341 ns
tpch_q21/DuckDB:duckdb 61822395 6.21548e+07 0.994652 ns
tpch_q22/DuckDB:duckdb 27037660 2.65073e+07 1.02001 ns

@github-actions
Copy link
Contributor

github-actions bot commented Oct 9, 2025 •
edited
Loading

Benchmarks: TPC-H SF=1 on S3

Summary

  • Overall: 1.000x ➖
  • Vortex: 0.991x ➖
  • duckdb:vortex: 0.988x ➖
  • datafusion:vortex: 0.995x ➖
  • Best: tpch_q09/DataFusion:vortex-compact (0.635x)
  • Worst: tpch_q08/DataFusion:vortex-file-compressed (1.503x)
  • Significant (>30%): 1↑ 1↓
Detailed Results Table
name PR daa3bf0 base d711afb ratio (PR/base) unit remark
tpch_q01/DataFusion:parquet 244879209 2.4869e+08 0.984678 ns
tpch_q02/DataFusion:parquet 371330733 3.68048e+08 1.00892 ns
tpch_q03/DataFusion:parquet 297655223 2.78445e+08 1.06899 ns
tpch_q04/DataFusion:parquet 137855691 1.37381e+08 1.00345 ns
tpch_q05/DataFusion:parquet 362458828 3.69891e+08 0.979907 ns
tpch_q06/DataFusion:parquet 128037414 1.2747e+08 1.00445 ns
tpch_q07/DataFusion:parquet 403151883 3.98244e+08 1.01232 ns
tpch_q08/DataFusion:parquet 452262789 4.6744e+08 0.967531 ns
tpch_q09/DataFusion:parquet 433637245 4.45815e+08 0.972685 ns
tpch_q10/DataFusion:parquet 443372682 4.52617e+08 0.979577 ns
tpch_q11/DataFusion:parquet 288733266 2.8899e+08 0.99911 ns
tpch_q12/DataFusion:parquet 210301534 2.17279e+08 0.967889 ns
tpch_q13/DataFusion:parquet 432068228 4.20909e+08 1.02651 ns
tpch_q14/DataFusion:parquet 153975970 1.54296e+08 0.997928 ns
tpch_q15/DataFusion:parquet 271667383 2.7198e+08 0.998849 ns
tpch_q16/DataFusion:parquet 185370344 1.81951e+08 1.01879 ns
tpch_q17/DataFusion:parquet 329886535 3.43959e+08 0.959086 ns
tpch_q18/DataFusion:parquet 411216810 4.13099e+08 0.995443 ns
tpch_q19/DataFusion:parquet 305837826 2.8986e+08 1.05512 ns
tpch_q20/DataFusion:parquet 264244406 2.65274e+08 0.996118 ns
tpch_q21/DataFusion:parquet 399420593 4.48157e+08 0.891252 ns
tpch_q22/DataFusion:parquet 169176389 1.70306e+08 0.993365 ns
tpch_q01/DataFusion:vortex-file-compressed 173599678 1.57683e+08 1.10094 ns
tpch_q02/DataFusion:vortex-file-compressed 201947413 1.97807e+08 1.02093 ns
tpch_q03/DataFusion:vortex-file-compressed 277562587 3.15877e+08 0.878705 ns
tpch_q04/DataFusion:vortex-file-compressed 154132800 1.38408e+08 1.11362 ns
tpch_q05/DataFusion:vortex-file-compressed 210782115 2.14985e+08 0.980453 ns
tpch_q06/DataFusion:vortex-file-compressed 154463269 1.49514e+08 1.0331 ns
tpch_q07/DataFusion:vortex-file-compressed 193795899 1.89119e+08 1.02473 ns
tpch_q08/DataFusion:vortex-file-compressed 436672233 2.90609e+08 1.50261 ns 🚨
tpch_q09/DataFusion:vortex-file-compressed 427950380 4.13852e+08 1.03407 ns
tpch_q10/DataFusion:vortex-file-compressed 249358484 2.44265e+08 1.02085 ns
tpch_q11/DataFusion:vortex-file-compressed 86426042 1.09457e+08 0.78959 ns
tpch_q12/DataFusion:vortex-file-compressed 217873634 2.15305e+08 1.01193 ns
tpch_q13/DataFusion:vortex-file-compressed 232338817 2.31588e+08 1.00324 ns
tpch_q14/DataFusion:vortex-file-compressed 108736502 1.1281e+08 0.963888 ns
tpch_q15/DataFusion:vortex-file-compressed 173403204 1.94781e+08 0.890245 ns
tpch_q16/DataFusion:vortex-file-compressed 181123889 1.87273e+08 0.967164 ns
tpch_q17/DataFusion:vortex-file-compressed 283783556 2.79614e+08 1.01491 ns
tpch_q18/DataFusion:vortex-file-compressed 217040406 2.28093e+08 0.951544 ns
tpch_q19/DataFusion:vortex-file-compressed 288925432 2.74098e+08 1.0541 ns
tpch_q20/DataFusion:vortex-file-compressed 184726988 1.89586e+08 0.974368 ns
tpch_q21/DataFusion:vortex-file-compressed 315352931 3.35719e+08 0.939335 ns
tpch_q22/DataFusion:vortex-file-compressed 147686028 1.48403e+08 0.995166 ns
tpch_q01/DataFusion:vortex-compact 153637751 1.75414e+08 0.875859 ns
tpch_q02/DataFusion:vortex-compact 189660972 1.98661e+08 0.954696 ns
tpch_q03/DataFusion:vortex-compact 293999625 2.88783e+08 1.01806 ns
tpch_q04/DataFusion:vortex-compact 175372089 1.69599e+08 1.03404 ns
tpch_q05/DataFusion:vortex-compact 190232863 1.80336e+08 1.05488 ns
tpch_q06/DataFusion:vortex-compact 178262181 1.72848e+08 1.03132 ns
tpch_q07/DataFusion:vortex-compact 180813584 1.8833e+08 0.960087 ns
tpch_q08/DataFusion:vortex-compact 438404250 3.50919e+08 1.2493 ns
tpch_q09/DataFusion:vortex-compact 248495063 3.91104e+08 0.635368 ns 🚀
tpch_q10/DataFusion:vortex-compact 267959868 2.72469e+08 0.983449 ns
tpch_q11/DataFusion:vortex-compact 90142780 8.63538e+07 1.04388 ns
tpch_q12/DataFusion:vortex-compact 274229595 2.68244e+08 1.02231 ns
tpch_q13/DataFusion:vortex-compact 177183746 1.81326e+08 0.977157 ns
tpch_q14/DataFusion:vortex-compact 133636510 1.32479e+08 1.00874 ns
tpch_q15/DataFusion:vortex-compact 198213025 2.18141e+08 0.908645 ns
tpch_q16/DataFusion:vortex-compact 196295592 1.90271e+08 1.03167 ns
tpch_q17/DataFusion:vortex-compact 301933500 2.64615e+08 1.14103 ns
tpch_q18/DataFusion:vortex-compact 209500145 2.28056e+08 0.918636 ns
tpch_q19/DataFusion:vortex-compact 329509278 3.34674e+08 0.984568 ns
tpch_q20/DataFusion:vortex-compact 181284736 2.06418e+08 0.878243 ns
tpch_q21/DataFusion:vortex-compact 365877170 3.67053e+08 0.996796 ns
tpch_q22/DataFusion:vortex-compact 148267843 1.34017e+08 1.10634 ns
tpch_q01/DuckDB:parquet 445146067 4.18803e+08 1.0629 ns
tpch_q02/DuckDB:parquet 1008493989 9.56537e+08 1.05432 ns
tpch_q03/DuckDB:parquet 740363651 6.78008e+08 1.09197 ns
tpch_q04/DuckDB:parquet 475590866 4.68428e+08 1.01529 ns
tpch_q05/DuckDB:parquet 1089118088 1.00456e+09 1.08417 ns
tpch_q06/DuckDB:parquet 312495354 2.79386e+08 1.11851 ns
tpch_q07/DuckDB:parquet 911269653 8.27942e+08 1.10064 ns
tpch_q08/DuckDB:parquet 1150070069 1.13424e+09 1.01396 ns
tpch_q09/DuckDB:parquet 1019058673 9.9875e+08 1.02033 ns
tpch_q10/DuckDB:parquet 996956112 9.11771e+08 1.09343 ns
tpch_q11/DuckDB:parquet 590609062 6.04605e+08 0.976851 ns
tpch_q12/DuckDB:parquet 429239816 4.19243e+08 1.02384 ns
tpch_q13/DuckDB:parquet 753224837 7.51822e+08 1.00187 ns
tpch_q14/DuckDB:parquet 473177376 4.53356e+08 1.04372 ns
tpch_q15/DuckDB:parquet 493791409 4.99014e+08 0.989535 ns
tpch_q16/DuckDB:parquet 325189430 3.67929e+08 0.883838 ns
tpch_q17/DuckDB:parquet 524806638 5.53775e+08 0.947689 ns
tpch_q18/DuckDB:parquet 618984373 6.39654e+08 0.967686 ns
tpch_q19/DuckDB:parquet 626329417 5.98616e+08 1.0463 ns
tpch_q20/DuckDB:parquet 761591740 7.16112e+08 1.06351 ns
tpch_q21/DuckDB:parquet 868566129 7.81611e+08 1.11125 ns
tpch_q22/DuckDB:parquet 370985918 3.15067e+08 1.17748 ns
tpch_q01/DuckDB:vortex-file-compressed 226446575 2.29726e+08 0.985723 ns
tpch_q02/DuckDB:vortex-file-compressed 767207410 8.20036e+08 0.935578 ns
tpch_q03/DuckDB:vortex-file-compressed 572250529 5.58777e+08 1.02411 ns
tpch_q04/DuckDB:vortex-file-compressed 321940756 3.41043e+08 0.943988 ns
tpch_q05/DuckDB:vortex-file-compressed 609298468 5.86107e+08 1.03957 ns
tpch_q06/DuckDB:vortex-file-compressed 290660020 3.03019e+08 0.959215 ns
tpch_q07/DuckDB:vortex-file-compressed 615759660 6.20846e+08 0.991807 ns
tpch_q08/DuckDB:vortex-file-compressed 812444706 8.38096e+08 0.969394 ns
tpch_q09/DuckDB:vortex-file-compressed 632348077 6.95132e+08 0.909681 ns
tpch_q10/DuckDB:vortex-file-compressed 579384793 6.07509e+08 0.953706 ns
tpch_q11/DuckDB:vortex-file-compressed 481051664 4.8288e+08 0.996214 ns
tpch_q12/DuckDB:vortex-file-compressed 404363587 4.024e+08 1.00488 ns
tpch_q13/DuckDB:vortex-file-compressed 274774979 2.83861e+08 0.967991 ns
tpch_q14/DuckDB:vortex-file-compressed 311910374 3.10313e+08 1.00515 ns
tpch_q15/DuckDB:vortex-file-compressed 237741222 2.41739e+08 0.983462 ns
tpch_q16/DuckDB:vortex-file-compressed 346427055 3.55268e+08 0.975116 ns
tpch_q17/DuckDB:vortex-file-compressed 474635626 4.74689e+08 0.999887 ns
tpch_q18/DuckDB:vortex-file-compressed 480445100 4.72107e+08 1.01766 ns
tpch_q19/DuckDB:vortex-file-compressed 392902709 3.82573e+08 1.027 ns
tpch_q20/DuckDB:vortex-file-compressed 543977037 5.3517e+08 1.01646 ns
tpch_q21/DuckDB:vortex-file-compressed 703006545 7.2893e+08 0.964436 ns
tpch_q22/DuckDB:vortex-file-compressed 276218754 2.79173e+08 0.989417 ns
tpch_q01/DuckDB:vortex-compact 222988961 2.33517e+08 0.954917 ns
tpch_q02/DuckDB:vortex-compact 761832955 7.9534e+08 0.957871 ns
tpch_q03/DuckDB:vortex-compact 562167787 5.66758e+08 0.991901 ns
tpch_q04/DuckDB:vortex-compact 325622591 3.10266e+08 1.04949 ns
tpch_q05/DuckDB:vortex-compact 599544354 6.08495e+08 0.985291 ns
tpch_q06/DuckDB:vortex-compact 287169736 2.8113e+08 1.02148 ns
tpch_q07/DuckDB:vortex-compact 563566724 5.99675e+08 0.939788 ns
tpch_q08/DuckDB:vortex-compact 787364659 8.21148e+08 0.958859 ns
tpch_q09/DuckDB:vortex-compact 631446754 6.04545e+08 1.0445 ns
tpch_q10/DuckDB:vortex-compact 542806493 5.71411e+08 0.94994 ns
tpch_q11/DuckDB:vortex-compact 447457652 4.68776e+08 0.954523 ns
tpch_q12/DuckDB:vortex-compact 392848710 3.98451e+08 0.98594 ns
tpch_q13/DuckDB:vortex-compact 254261347 2.60722e+08 0.975222 ns
tpch_q14/DuckDB:vortex-compact 303018371 2.87966e+08 1.05227 ns
tpch_q15/DuckDB:vortex-compact 257327493 2.34463e+08 1.09752 ns
tpch_q16/DuckDB:vortex-compact 332454626 3.56661e+08 0.932131 ns
tpch_q17/DuckDB:vortex-compact 456061758 4.80767e+08 0.948612 ns
tpch_q18/DuckDB:vortex-compact 425946277 4.34606e+08 0.980073 ns
tpch_q19/DuckDB:vortex-compact 378572421 3.90171e+08 0.970273 ns
tpch_q20/DuckDB:vortex-compact 548250013 5.46816e+08 1.00262 ns
tpch_q21/DuckDB:vortex-compact 738405674 6.81573e+08 1.08338 ns
tpch_q22/DuckDB:vortex-compact 270553382 2.64277e+08 1.02375 ns

@cloudflare-workers-and-pages
Copy link

cloudflare-workers-and-pages bot commented Oct 10, 2025 •
edited
Loading

Deploying vortex-bench with  Cloudflare Pages  Cloudflare Pages

Latest commit: aa9d87a
Status: ✅  Deploy successful!
Preview URL: https://ab56697b.vortex-93b.pages.dev
Branch Preview URL: https://ct-lance-compress.vortex-93b.pages.dev

View logs

@connortsui20 connortsui20 added the action/benchmark Trigger full benchmarks to run on this PR label Oct 10, 2025
@github-actions github-actions bot removed the action/benchmark Trigger full benchmarks to run on this PR label Oct 10, 2025
@connortsui20
refactor compression benchmarks
2a0e087 
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20 connortsui20 force-pushed the ct/lance-compress branch 6 times, most recently from 1d39ba5 to daa3bf0 Compare October 10, 2025 16:24
@connortsui20 connortsui20 added the action/benchmark Trigger full benchmarks to run on this PR label Oct 10, 2025
@github-actions github-actions bot removed the action/benchmark Trigger full benchmarks to run on this PR label Oct 10, 2025
@connortsui20
add lance compress benchmark
aa9d87a 
Additionally cleans up some other things.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20 connortsui20 merged commit f747121 into develop Oct 10, 2025
40 checks passed
@connortsui20 connortsui20 deleted the ct/lance-compress branch October 10, 2025 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@joseph-isaacs joseph-isaacs Awaiting requested review from joseph-isaacs

Assignees

No one assigned

Labels

changelog/feature A new feature

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@connortsui20 @joseph-isaacs