Skip to content

Comments

feat: reduce size of segments containing fixed-size-list arrays#6180

Merged
danking merged 8 commits intodevelopfrom
dk/better-segment-sizes-for-fixed-size-list
Jan 27, 2026
Merged

feat: reduce size of segments containing fixed-size-list arrays#6180
danking merged 8 commits intodevelopfrom
dk/better-segment-sizes-for-fixed-size-list

Conversation

@danking
Copy link
Contributor

@danking danking commented Jan 27, 2026

I also verified this on some fixed-size-list data I have locally and, indeed, I see more reasonable segment sizes (~1MiB).

I also verified this on some fixed-size-list data I have locally and, indeed, I see more reasonable
segment sizes (~1MiB).

Signed-off-by: Daniel King <dan@spiraldb.com>
@connortsui20 connortsui20 added the changelog/feature A new feature label Jan 27, 2026
@codspeed-hq
Copy link

codspeed-hq bot commented Jan 27, 2026

CodSpeed Performance Report

Merging this PR will degrade performance by 20.01%

Comparing dk/better-segment-sizes-for-fixed-size-list (ad235e9) with develop (83664a0)

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

Summary

⚡ 10 improved benchmarks
❌ 7 regressed benchmarks
✅ 1144 untouched benchmarks
⏩ 1323 skipped benchmarks1

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation canonical_into_non_nullable[(10000, 1, 0.0)] 36.2 µs 24.9 µs +45.52%
Simulation canonical_into_non_nullable[(10000, 1, 0.01)] 41 µs 31.3 µs +31.28%
Simulation canonical_into_non_nullable[(10000, 10, 0.0)] 278.8 µs 194.3 µs +43.52%
Simulation canonical_into_non_nullable[(10000, 1, 0.1)] 57 µs 47.1 µs +21.09%
Simulation canonical_into_non_nullable[(10000, 10, 0.01)] 306 µs 221.3 µs +38.31%
Simulation canonical_into_non_nullable[(10000, 10, 0.1)] 471.6 µs 380.7 µs +23.86%
Simulation canonical_into_nullable[(10000, 100, 0.0)] 4.3 ms 5 ms -12.44%
Simulation into_canonical_non_nullable[(10000, 10, 0.0)] 283.7 µs 201.7 µs +40.66%
Simulation into_canonical_non_nullable[(10000, 1, 0.01)] 38.3 µs 46.3 µs -17.25%
Simulation into_canonical_non_nullable[(10000, 1, 0.1)] 54.4 µs 62.8 µs -13.37%
Simulation into_canonical_non_nullable[(10000, 10, 0.1)] 473 µs 385.2 µs +22.81%
Simulation into_canonical_non_nullable[(10000, 10, 0.01)] 310.6 µs 229.2 µs +35.47%
Simulation into_canonical_non_nullable[(10000, 1, 0.0)] 32.3 µs 40.4 µs -20.01%
Simulation into_canonical_nullable[(10000, 100, 0.0)] 4.3 ms 5.1 ms -14.45%
Simulation into_canonical_nullable[(10000, 10, 0.0)] 459.1 µs 541.3 µs -15.2%
Simulation into_canonical_nullable[(10000, 10, 0.1)] 720.2 µs 632.2 µs +13.91%
Simulation into_canonical_nullable[(10000, 100, 0.1)] 6.1 ms 6.9 ms -12.01%

Footnotes

  1. 1323 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

// `write_stream` from combining adjacent 0.9 MiB chunks into one 1.8 MiB chunk.
let max_rows = usize::try_from(block_size_target.div_ceil(elem_size as u64))
.unwrap_or(usize::MAX);
self.block_len_multiple.min(max_rows).max(1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should help with large binary values too, no?

Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
@danking danking requested a review from connortsui20 January 27, 2026 21:49
Copy link
Contributor

@connortsui20 connortsui20 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice

Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
@danking danking merged commit 44619e4 into develop Jan 27, 2026
45 of 47 checks passed
@danking danking deleted the dk/better-segment-sizes-for-fixed-size-list branch January 27, 2026 22:51
danking added a commit that referenced this pull request Feb 6, 2026
I also verified this on some fixed-size-list data I have locally and,
indeed, I see more reasonable segment sizes (~1MiB).

---------

Signed-off-by: Daniel King <dan@spiraldb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants