Faster Pipelines

[gatesn]

Removes a bunch of the pipeline calling overhead.

[gatesn]

It's actually very useful to have highly specialized filter kernels for N elements. So I think we keep these here, outside the vortex-compute package, possibly delegating to vortex-compute where possible.

[codspeed-hq]

CodSpeed Performance Report

Merging #5352 will degrade performances by 25.7%

_{Comparing ngates/faster-pipelines (b925b5d) with develop (8f7b707)}

Summary

⚡ 2 improvements
❌ 8 regressions
✅ 1400 untouched
🆕 7 new
⏩ 624 skipped¹
🗄️ 28 archived benchmarks run²

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	BASE	HEAD	Change
⚡	bitpack_pipeline_unpack[(10000, 0.5)]	90.9 µs	80.3 µs	+13.15%
⚡	bitpack_pipeline_unpack[(100000, 0.5)]	497.6 µs	392.1 µs	+26.92%
❌	filter_runend[(1000, 256, 0.005)]	19.9 µs	22.3 µs	-10.61%
❌	filter_runend[(1000, 256, 0.01)]	20.8 µs	23.2 µs	-10.19%
❌	filter_runend[(1000, 256, 0.03)]	20 µs	22.4 µs	-10.57%
❌	slice_vortex_buffer[1024]	2.4 µs	3 µs	-20.58%
❌	slice_vortex_buffer[128]	1.8 µs	2.4 µs	-25.7%
❌	slice_vortex_buffer[16384]	2.5 µs	3.2 µs	-19.35%
❌	slice_vortex_buffer[2048]	2.3 µs	2.9 µs	-20.78%
❌	slice_vortex_buffer[65536]	2.7 µs	3.3 µs	-18.46%
🆕	decompress[("alp_for_bp_f64", 0x4639b20)]	N/A	24.2 ms	N/A
🆕	decompress[("datetime_for_bp", 0x463c9e0)]	N/A	34.9 ms	N/A
🆕	decompress[("dict_fsst_varbin_bp_string", 0x463bd20)]	N/A	14.5 ms	N/A
🆕	decompress[("dict_fsst_varbin_string", 0x463b880)]	N/A	14.5 ms	N/A
🆕	decompress[("dict_varbinview_string", 0x463a540)]	N/A	14.7 ms	N/A
🆕	decompress[("for_bp_u64", 0x46393d0)]	N/A	2.5 ms	N/A
🆕	decompress[("runend_for_bp_u32", 0x463a9d0)]	N/A	2 ms	N/A

Footnotes

1. 624 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. 28 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩

[codecov]

Codecov Report

❌ Patch coverage is 61.42384% with 233 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.90%. Comparing base (cbf93fd) to head (b925b5d).
⚠️ Report is 6 commits behind head on develop.

Files with missing lines	Patch %	Lines
vortex-vector/src/primitive/vector.rs	0.00%	66 Missing ⚠️
vortex-vector/src/vector.rs	0.00%	48 Missing ⚠️
vortex-compute/src/filter/slice_mut.rs	53.03%	31 Missing ⚠️
vortex-compute/src/filter/mask.rs	0.00%	16 Missing ⚠️
vortex-array/src/pipeline/driver/mod.rs	57.14%	12 Missing ⚠️
vortex-compute/src/filter/vector/mod.rs	0.00%	12 Missing ⚠️
vortex-mask/src/mask_mut.rs	0.00%	8 Missing ⚠️
vortex-compute/src/filter/buffer.rs	58.82%	7 Missing ⚠️
...fastlanes/src/bitpacking/array/bitpack_pipeline.rs	87.17%	5 Missing ⚠️
vortex-buffer/src/bit/buf.rs	50.00%	5 Missing ⚠️
... and 8 more

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[fargito]

Hello, CodSpeed team member here.

We had some trouble processing the results in your latest run, due to the large number of uploaded benchmarks. We have some limitations on our side that we are currently working on. In the meantime, could you consider sharding the benchmarks into more than 2 jobs? It should fix the issue for now.

Here are some docs on benchmark sharding.