Skip to content

Conversation

@danking
Copy link
Contributor

@danking danking commented Sep 8, 2025

For TPC-H, this brings the size from larger than Vortex standard to smaller.

Compressor Size (GiB) Size (btrblocks = 1.0)
btrblocks 2.9 1.00
compact (develop) 3.7 1.28
compact (this PR) 2.2 0.76

This has no effect on benchmarks run on my laptop.

I also avoided returning pco/zstd compressed arrays which were bigger than the uncompressed version. I saw this happen on some TPC-H columns. I suspec there's a real benefit to setting some size goal, like 90%, and storing uncompressed data unless we meet that goal.

For TPC-H, this brings the size from larger than Vortex standard to smaller.

|        Compressor  | Size (GiB) | Size (btrblocks = 1.0) |
| ------------------ | ---------- | ---------------------- |
|          btrblocks |        2.9 |                   1.00 |
|  compact (develop) |        3.7 |                   1.28 |
|  compact (this PR) |        2.2 |                   0.76 |

This has no effect on benchmarks run on my laptop.

I also avoided returning pco/zstd compressed arrays which were bigger than the uncompressed
version. I saw this happen on some TPC-H columns. I suspec there's a real benefit to setting some
size goal, like 90%, and storing uncompressed data unless we meet that goal.

Signed-off-by: Daniel King <dan@spiraldb.com>
@danking danking requested review from a10y and robert3005 September 8, 2025 16:32
@danking danking marked this pull request as ready for review September 8, 2025 16:32
Signed-off-by: Daniel King <dan@spiraldb.com>
@codspeed-hq
Copy link

codspeed-hq bot commented Sep 8, 2025

CodSpeed Performance Report

Merging #4569 will not alter performance

Comparing dk/vx-compact-compresses-decimal (6ece637) with develop (60775e5)

Summary

✅ 1351 untouched benchmarks

@danking danking added the changelog/feature A new feature label Sep 8, 2025
Signed-off-by: Daniel King <dan@spiraldb.com>
@codecov
Copy link

codecov bot commented Sep 8, 2025

Codecov Report

❌ Patch coverage is 70.83333% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.57%. Comparing base (60775e5) to head (6ece637).
⚠️ Report is 2 commits behind head on develop.

Files with missing lines Patch % Lines
vortex-layout/src/layouts/compact.rs 67.56% 12 Missing ⚠️
vortex-array/src/arrays/decimal/narrow.rs 81.81% 2 Missing ⚠️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@danking danking merged commit 0036b15 into develop Sep 8, 2025
39 checks passed
@danking danking deleted the dk/vx-compact-compresses-decimal branch September 8, 2025 17:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants