Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add TakeOptions to skip bounds checking #1343

Merged
merged 4 commits into from
Nov 17, 2024
Merged

Conversation

gatesn
Copy link
Contributor

@gatesn gatesn commented Nov 17, 2024

For Q6 and Q19 this seems to help by ~15%. For the others, it's reported as an increase, but I'm not convinced this isn't just a slow benchmark run since there shouldn't be any reason for an increase.

@gatesn gatesn added the benchmark Run benchmarks on this branch label Nov 17, 2024
@github-actions github-actions bot removed the benchmark Run benchmarks on this branch label Nov 17, 2024
@gatesn gatesn added the benchmark Run benchmarks on this branch label Nov 17, 2024
@github-actions github-actions bot removed benchmark Run benchmarks on this branch labels Nov 17, 2024
@gatesn gatesn changed the title Add unchecked take option Add TakeOptions to skip bounds checking Nov 17, 2024
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Vortex bytes_at

Benchmark suite Current: bdf077b Previous: ff6c440 Ratio
bytes_at/array_data 1101.1459341949742 ns (3.285858867570596) 1042.1470969126294 ns (2.7500252515723105) 1.06
bytes_at/array_view 899.9561726782679 ns (2.142503635016965) 835.7179145365357 ns (1.7966804160627703) 1.08

This comment was automatically generated by workflow using github-action-benchmark.

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DataFusion

Benchmark suite Current: bdf077b Previous: ff6c440 Ratio
arrow/planning 817222.2762724921 ns (2455.21357242472) 841203.4326846935 ns (23550.597091191798) 0.97
arrow/exec 1421437.56566905 ns (4841.038651178125) 1422202.6033632478 ns (8087.174718600581) 1.00
vortex-pushdown-compressed/planning 502998.49039800355 ns (799.6147233224765) 503810.0389279283 ns (1017.1571389370074) 1.00
vortex-pushdown-compressed/exec 2307953.393636363 ns (7562.187568182824) 2261120.728695652 ns (9010.307141303318) 1.02
vortex-pushdown-uncompressed/planning 501809.76986909355 ns (1136.9554979604727) 504480.17604964704 ns (1119.2507154082414) 0.99
vortex-pushdown-uncompressed/exec 1184985.8692772314 ns (4918.5857243106) 1151323.0589950487 ns (6188.314262179309) 1.03
vortex-nopushdown-compressed/planning 829887.562926642 ns (1721.7185667120502) 828046.8376967842 ns (1839.0946128218784) 1.00
vortex-nopushdown-compressed/exec 3393362.536875 ns (45640.58767187502) 2978947.8123529423 ns (7236.443250000011) 1.14
vortex-nopushdown-uncompressed/planning 825705.5934650454 ns (1291.2186877781642) 827504.3093578133 ns (1569.1447213808424) 1.00
vortex-nopushdown-uncompressed/exec 6250497.674444445 ns (137307.25352777727) 4524610.137500002 ns (9629.796072916128) 1.38

This comment was automatically generated by workflow using github-action-benchmark.

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Random Access

Benchmark suite Current: bdf077b Previous: ff6c440 Ratio
random-access/vortex-tokio-local-disk 2199535.3578260858 ns (16295.053706521401) 2102985.206666667 ns (9328.485208333237) 1.05
random-access/vortex-local-fs 2843962.2755555566 ns (37671.04050694476) 2797069.024210526 ns (11268.978243421298) 1.02
random-access/parquet-tokio-local-disk 222397331.9333333 ns (5365543.098333344) 211491204.73333335 ns (2682954.9662500173) 1.05

This comment was automatically generated by workflow using github-action-benchmark.

@gatesn gatesn enabled auto-merge (squash) November 17, 2024 18:00
@gatesn gatesn disabled auto-merge November 17, 2024 18:00
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TPC-H

Benchmark suite Current: bdf077b Previous: ff6c440 Ratio
tpch_q1/vortex-in-memory-pushdown 456989635 ns (2559625.349999994) 444338034.1 ns (1273099.7000000179) 1.03
tpch_q1/arrow 551824906.6 ns (6541537.433750033) 526315917.5 ns (2076435.0712500215) 1.05
tpch_q1/parquet 755737410.2 ns (7715642.701250017) 727571111.8 ns (1539656.3087500334) 1.04
tpch_q1/vortex-file-compressed 502197989.35 ns (5683635.550000012) 481785173.1 ns (1937024.5368749797) 1.04
tpch_q2/vortex-in-memory-pushdown 124606985.15833333 ns (2349597.2546875104) 117299885.3847619 ns (409951.83432142437) 1.06
tpch_q2/arrow 121926947.83285718 ns (1953588.746616073) 115009280.92075396 ns (832434.3822197542) 1.06
tpch_q2/parquet 149345472.22599205 ns (924060.7121627033) 144552738.88829362 ns (709022.4526364058) 1.03
tpch_q2/vortex-file-compressed 169027339.55039683 ns (653967.6271428615) 149229868.14373016 ns (1131945.397747025) 1.13
tpch_q3/vortex-in-memory-pushdown 179934020.3333333 ns (1095123.5499999821) 170487690.43551588 ns (765003.5345064402) 1.06
tpch_q3/arrow 146121432.0486111 ns (763490.3263229281) 143402328.9613492 ns (1136899.5467122942) 1.02
tpch_q3/parquet 337763231.3 ns (1560253.5443750322) 326633493.55 ns (1676046.224999994) 1.03
tpch_q3/vortex-file-compressed 245280102.79999995 ns (1818326.3466666192) 233614270.3333333 ns (1353445.8895833343) 1.05
tpch_q4/vortex-in-memory-pushdown 131365543.96357143 ns (568797.3615178615) 125911742.1372619 ns (536476.2028645724) 1.04
tpch_q4/arrow 183011633.6666667 ns (659328.4416666776) 181132071.88039678 ns (615144.6329742074) 1.01
tpch_q4/parquet 195399701.56666666 ns (795170.5479166508) 194341809.8333333 ns (804782.025000006) 1.01
tpch_q4/vortex-file-compressed 196229728.76666668 ns (733035.9145833254) 188037739.40000004 ns (568832.295416668) 1.04
tpch_q5/vortex-in-memory-pushdown 289181428.25 ns (5180233.474999994) 276755448.5 ns (2050493.900000006) 1.04
tpch_q5/arrow 269176007.1 ns (2774202.0381249934) 256291452.4 ns (1735672.3087500036) 1.05
tpch_q5/parquet 427162518.2 ns (3243705.1368750036) 422962599.55 ns (2689625.467500001) 1.01
tpch_q5/vortex-file-compressed 329600437.1 ns (5497144.826875001) 311912716.8 ns (3125630.4399999976) 1.06
tpch_q6/vortex-in-memory-pushdown 56898516.09345238 ns (311469.8675000034) 54268390.6283135 ns (267063.50343056023) 1.05
tpch_q6/arrow 27031036.113323413 ns (137175.1652046144) 25614231.187896825 ns (195042.0080451388) 1.06
tpch_q6/parquet 137892663.33813494 ns (648506.3212896883) 136450846.4170238 ns (384801.1989494115) 1.01
tpch_q6/vortex-file-compressed 11105024.611124339 ns (269994.0239971895) 12180742.200436508 ns (283484.08135342225) 0.91
tpch_q7/vortex-in-memory-pushdown 578540389.2 ns (4390762.652500033) 552794634.7 ns (3077859.09375) 1.05
tpch_q7/arrow 536802868.6 ns (4347290.120000005) 517568937.9 ns (2644139.599999994) 1.04
tpch_q7/parquet 688977775.2 ns (14238112.50999999) 638566835.3 ns (3558171.011250019) 1.08
tpch_q7/vortex-file-compressed 651288659.6 ns (5565398.75) 606340759.4 ns (2937459.8450000286) 1.07
tpch_q8/vortex-in-memory-pushdown 235110938.3333333 ns (2503917.0520833433) 225313654.53333336 ns (1383129.0791666508) 1.04
tpch_q8/arrow 210059282.2 ns (1581205.8479166627) 205275994.56666666 ns (1302647.6424999833) 1.02
tpch_q8/parquet 480542892.8 ns (2470600.1487499774) 472345313.9 ns (2291731.7068749964) 1.02
tpch_q8/vortex-file-compressed 287250453.05 ns (2660972.618750006) 277508578.45 ns (1896450.1443750262) 1.04
tpch_q9/vortex-in-memory-pushdown 417999611.4 ns (5057854.428124994) 399096652.3 ns (3796922.071875006) 1.05
tpch_q9/arrow 398867018.35 ns (2878056.8950000107) 382510568 ns (2638162.6387500167) 1.04
tpch_q9/parquet 707444958.6 ns (2793397.1799999475) 682263729.9 ns (4536992.589999974) 1.04
tpch_q9/vortex-file-compressed 475738446.45 ns (8101726.050000012) 446133790.25 ns (3693823.0400000215) 1.07
tpch_q10/vortex-in-memory-pushdown 289680776.55 ns (1519974.7843749821) 277631683 ns (1241113.150000006) 1.04
tpch_q10/arrow 257682124.6 ns (1305791.3037500083) 237754551.9333333 ns (1481243.5270833224) 1.08
tpch_q10/parquet 491261904.55 ns (4031380.463750005) 476331239.65 ns (2267864.233125001) 1.03
tpch_q10/vortex-file-compressed 303804280.65 ns (1492066.7431250215) 296456714.05 ns (2343033.510625005) 1.02
tpch_q11/vortex-in-memory-pushdown 116169427.48333332 ns (448079.4751041755) 114132449.19035713 ns (474950.05083332956) 1.02
tpch_q11/arrow 115607447.79420635 ns (2327877.6802777797) 109452851.27833334 ns (801196.5098124966) 1.06
tpch_q11/parquet 125165234.61845239 ns (1338748.3850044683) 118874173.3704762 ns (437316.7145714313) 1.05
tpch_q11/vortex-file-compressed 212932264.3 ns (3447113.850000009) 188798383.5 ns (1125463.934583336) 1.13
tpch_q12/vortex-in-memory-pushdown 244885354.1 ns (578132.1570833325) 239741668.16666666 ns (841440.739166677) 1.02
tpch_q12/arrow 188787359.16666666 ns (493578.8833333254) 186456298.93333334 ns (433295.48041665554) 1.01
tpch_q12/parquet 314559270.05 ns (679588.4074999988) 311642382.25 ns (554267.9631249905) 1.01
tpch_q12/vortex-file-compressed 203327211.23333332 ns (728323.8816666752) 206564585.6 ns (1522454.0499999821) 0.98
tpch_q13/vortex-in-memory-pushdown 188455419.85214287 ns (3118378.23368752) 162992629.0948016 ns (1200647.2813080251) 1.16
tpch_q13/arrow 179739605.33960316 ns (1427550.3204761744) 157121486.82190478 ns (1708222.3491815627) 1.14
tpch_q13/parquet 334310570.1 ns (2417137.699999988) 298731987.45 ns (1116557.375) 1.12
tpch_q13/vortex-file-compressed 198835003.9 ns (2287369.4975000024) 173896084.2672619 ns (1467701.9288987964) 1.14
tpch_q14/vortex-in-memory-pushdown 71822346.74954365 ns (453447.13861110806) 67892092.51976189 ns (272854.90418154746) 1.06
tpch_q14/arrow 36481923.5825926 ns (258640.99453703687) 33753661.027354494 ns (635360.4032010604) 1.08
tpch_q14/parquet 220398919.73333335 ns (678412.7541666925) 219782088.0666667 ns (606545.7462500185) 1.00
tpch_q14/vortex-file-compressed 94747108.32416667 ns (905257.1779479012) 87746368.37021825 ns (1829163.5712326318) 1.08
tpch_q15/vortex-in-memory-pushdown 99703385.11218253 ns (824442.3663690537) 93069322.57952382 ns (926686.3283809498) 1.07
tpch_q15/arrow 59423338.49386905 ns (805727.2081845216) 57347584.716269836 ns (399781.35846428946) 1.04
tpch_q15/parquet 302097596.9 ns (1887899.0806250274) 297247353.35 ns (1451596.7862499952) 1.02
tpch_q15/vortex-file-compressed 171603446.77507937 ns (2102699.904801592) 173673562.69674602 ns (4206420.370200381) 0.99
tpch_q16/vortex-in-memory-pushdown 100933988.6881746 ns (760695.7262728289) 96190209.21123016 ns (649848.5485491082) 1.05
tpch_q16/arrow 89247685.82869047 ns (705962.2319613025) 83330982.11505952 ns (352169.27984374017) 1.07
tpch_q16/parquet 102760430.99976191 ns (576726.8201785684) 97336342.1895635 ns (697194.0600704402) 1.06
tpch_q16/vortex-file-compressed 107576264.37246032 ns (743845.1499206349) 102470096.58369048 ns (584575.1019270867) 1.05
tpch_q17/vortex-in-memory-pushdown 635136614 ns (13697584.891250014) 570049459 ns (5101091.133749962) 1.11
tpch_q17/arrow 556206947.7 ns (4238241.125) 471278395.95 ns (5309915.824999988) 1.18
tpch_q17/parquet 643039834.1 ns (4443720.901250005) 625618786.2 ns (3950672.3125) 1.03
tpch_q17/vortex-file-compressed 619680667.7 ns (5549139.662500024) 575875429.1 ns (8372785.633749962) 1.08
tpch_q18/vortex-in-memory-pushdown 1089226618.6 ns (13998157.459999919) 1015932790 ns (7217276.081250012) 1.07
tpch_q18/arrow 1043378413 ns (3860243.245000005) 1009033363.4 ns (5212146.94749999) 1.03
tpch_q18/parquet 1317132788.8 ns (29321027.98124993) 1199417814.5 ns (7754283.473750114) 1.10
tpch_q18/vortex-file-compressed 1152685561.2 ns (14863832.210000038) 1077707581 ns (9238559.142500103) 1.07
tpch_q19/vortex-in-memory-pushdown 243886621.96666664 ns (1047482.4191666842) 237959345.03333336 ns (522513.0479166508) 1.02
tpch_q19/arrow 167946044.7856746 ns (448028.0061607063) 166240961.94083333 ns (386609.2526771128) 1.01
tpch_q19/parquet 419303481.85 ns (1870495.2749999762) 417631060.35 ns (2367817.574999988) 1.00
tpch_q19/vortex-file-compressed 173863252.31055555 ns (3027317.727222204) 203492220.93333334 ns (3407535.9166666716) 0.85
tpch_q20/vortex-in-memory-pushdown 250109316.5 ns (6690169.100000009) 215840421.96666664 ns (2028237.599999994) 1.16
tpch_q20/arrow 225701032.3 ns (2751526.235000029) 193702813.06666666 ns (2167096.0883333385) 1.17
tpch_q20/parquet 331312172.2 ns (2637748.9962500036) 314295612.3 ns (2126816.854375005) 1.05
tpch_q20/vortex-file-compressed 303284222.6 ns (2350712.523750007) 274327324.15 ns (3077051.592500001) 1.11
tpch_q21/vortex-in-memory-pushdown 936422112.6 ns (6543649.029999971) 878704229.3 ns (4672984.847500026) 1.07
tpch_q21/arrow 882150007.8 ns (9414630.389999986) 841966375.1 ns (5322940.304999948) 1.05
tpch_q21/parquet 980871886.5 ns (3251130.1887500286) 913994725.4 ns (5523395.70875001) 1.07
tpch_q21/vortex-file-compressed 1039424891 ns (6904593.375) 1009582744.5 ns (5683014.477499962) 1.03
tpch_q22/vortex-in-memory-pushdown 76904082.04458334 ns (268687.8611666709) 75578285.3994246 ns (317960.4942626506) 1.02
tpch_q22/arrow 76658255.43287697 ns (460472.1141837761) 73899776.3859127 ns (197508.04556645453) 1.04
tpch_q22/parquet 104402024.63615079 ns (607266.3912033811) 102542421.70587303 ns (335852.07493849844) 1.02
tpch_q22/vortex-file-compressed 83280018.68819444 ns (674975.3209722191) 81352453.38111112 ns (483472.48770486563) 1.02

This comment was automatically generated by workflow using github-action-benchmark.

@gatesn gatesn enabled auto-merge (squash) November 17, 2024 18:32
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Vortex Compression

Benchmark suite Current: bdf077b Previous: ff6c440 Ratio
compress time/taxi 1297095862.2 ns (2091105) 1290392384.8 ns (1543331.1000000238) 1.01
compress time/taxi throughput 470808924 bytes 470808924 bytes 1
parquet_rs-zstd compress time/taxi 1785702046.1 ns (2253886.1012499332) 1759494926.3 ns (3801543.149999976) 1.01
parquet_rs-zstd compress time/taxi throughput 470808924 bytes 470808924 bytes 1
decompress time/taxi 428107458.75 ns (1418312.025000006) 425697608.2 ns (2915230.725000024) 1.01
decompress time/taxi throughput 470808924 bytes 470808924 bytes 1
parquet_rs-zstd decompress time/taxi 308763790.25 ns (826168.6475000083) 309661469.7 ns (716313.6543749869) 1.00
parquet_rs-zstd decompress time/taxi throughput 470808924 bytes 470808924 bytes 1
vortex:parquet-zstd size/taxi 0.9861801500014028 ratio 0.9862522004238323 ratio 1.00
vortex:raw size/taxi 0.11721888262253924 ratio 0.11722690286133998 ratio 1.00
vortex size/taxi 55187696 bytes 55191472 bytes 1.00
compress time/AirlineSentiment 907391.414711508 ns (966.609641597257) 902512.3408498351 ns (1358.1536325289053) 1.01
compress time/AirlineSentiment throughput 2020 bytes 2020 bytes 1
parquet_rs-zstd compress time/AirlineSentiment 55908.72140642867 ns (127.68778099986594) 55912.48456745787 ns (150.394085110609) 1.00
parquet_rs-zstd compress time/AirlineSentiment throughput 2020 bytes 2020 bytes 1
decompress time/AirlineSentiment 117413.34516308243 ns (852.4722250268751) 114235.86890268713 ns (531.5000359434125) 1.03
decompress time/AirlineSentiment throughput 2020 bytes 2020 bytes 1
parquet_rs-zstd decompress time/AirlineSentiment 32488.857251518435 ns (50.62959661788045) 32672.897035922088 ns (95.52898248267957) 0.99
parquet_rs-zstd decompress time/AirlineSentiment throughput 2020 bytes 2020 bytes 1
vortex:parquet-zstd size/AirlineSentiment 9.294725956566701 ratio 9.294725956566701 ratio 1
vortex:raw size/AirlineSentiment 4.44950495049505 ratio 4.44950495049505 ratio 1
vortex size/AirlineSentiment 8988 bytes 8988 bytes 1
compress time/Arade 3000951648.1 ns (3477911.071249962) 2979935172.9 ns (2973815.1599998474) 1.01
compress time/Arade throughput 787023760 bytes 787023760 bytes 1
parquet_rs-zstd compress time/Arade 3080089830 ns (6292928.342499971) 3041426467.6 ns (5365352.806250095) 1.01
parquet_rs-zstd compress time/Arade throughput 787023760 bytes 787023760 bytes 1
decompress time/Arade 776755202.8 ns (4121922.1474999785) 846467967.1 ns (3516144.100000024) 0.92
decompress time/Arade throughput 787023760 bytes 787023760 bytes 1
parquet_rs-zstd decompress time/Arade 656427632.3 ns (3151713.300000012) 658328747.6 ns (2368142.121249974) 1.00
parquet_rs-zstd decompress time/Arade throughput 787023760 bytes 787023760 bytes 1
vortex:parquet-zstd size/Arade 0.48886972728589 ratio 0.48887014641775883 ratio 1.00
vortex:raw size/Arade 0.1896989641075131 ratio 0.1896989641075131 ratio 1
vortex size/Arade 149297592 bytes 149297592 bytes 1
compress time/Bimbo 9903586172.2 ns (8964662.907500267) 10706613668.1 ns (12047749.274999619) 0.92
compress time/Bimbo throughput 7121333608 bytes 7121333608 bytes 1
parquet_rs-zstd compress time/Bimbo 21187090768.5 ns (33780624.75) 20332296368.1 ns (48459356.17374992) 1.04
parquet_rs-zstd compress time/Bimbo throughput 7121333608 bytes 7121333608 bytes 1
decompress time/Bimbo 4609657523.3 ns (17892630.374999523) 4993835622 ns (10009293.599999905) 0.92
decompress time/Bimbo throughput 7121333608 bytes 7121333608 bytes 1
parquet_rs-zstd decompress time/Bimbo 2621525046.6 ns (3179039.754999876) 2628970679.1 ns (6439381.452500105) 1.00
parquet_rs-zstd decompress time/Bimbo throughput 7121333608 bytes 7121333608 bytes 1
vortex:parquet-zstd size/Bimbo 1.3367483791165071 ratio 1.335461457174769 ratio 1.00
vortex:raw size/Bimbo 0.07285997378596619 ratio 0.07278982962091277 ratio 1.00
vortex size/Bimbo 518860180 bytes 518360660 bytes 1.00
compress time/CMSprovider 15961847332.1 ns (24771005.863749504) 16207682746.7 ns (27983609.16250038) 0.98
compress time/CMSprovider throughput 5149123964 bytes 5149123964 bytes 1
parquet_rs-zstd compress time/CMSprovider 20303819361.3 ns (20076737.950000763) 20459914140.5 ns (60863766.04999924) 0.99
parquet_rs-zstd compress time/CMSprovider throughput 5149123964 bytes 5149123964 bytes 1
decompress time/CMSprovider 5964221620.2 ns (23788380.25) 6050584640.8 ns (45585286.37125015) 0.99
decompress time/CMSprovider throughput 5149123964 bytes 5149123964 bytes 1
parquet_rs-zstd decompress time/CMSprovider 5123652610.9 ns (13161863.862499714) 5148552841.6 ns (14231531.05000019) 1.00
parquet_rs-zstd decompress time/CMSprovider throughput 5149123964 bytes 5149123964 bytes 1
vortex:parquet-zstd size/CMSprovider 1.2251309982595753 ratio 1.2355169348599941 ratio 0.99
vortex:raw size/CMSprovider 0.18309051920118039 ratio 0.1846428018915724 ratio 0.99
vortex size/CMSprovider 942755780 bytes 950748676 bytes 0.99
compress time/Euro2016 2962142912.3 ns (4938397.717499971) 2991626287.8 ns (2001795.551249981) 0.99
compress time/Euro2016 throughput 393253221 bytes 393253221 bytes 1
parquet_rs-zstd compress time/Euro2016 1574840952.4 ns (3420989.9962500334) 1568879088 ns (3142854.133749962) 1.00
parquet_rs-zstd compress time/Euro2016 throughput 393253221 bytes 393253221 bytes 1
decompress time/Euro2016 340606522.1 ns (1303537.053124994) 339156585.85 ns (2085657.1474999785) 1.00
decompress time/Euro2016 throughput 393253221 bytes 393253221 bytes 1
parquet_rs-zstd decompress time/Euro2016 483980952.85 ns (1038351.6499999762) 491633741.5 ns (2719679.90625) 0.98
parquet_rs-zstd decompress time/Euro2016 throughput 393253221 bytes 393253221 bytes 1
vortex:parquet-zstd size/Euro2016 1.4442742122387626 ratio 1.4443226604273025 ratio 1.00
vortex:raw size/Euro2016 0.43663878343669055 ratio 0.4366534304877315 ratio 1.00
vortex size/Euro2016 171709608 bytes 171715368 bytes 1.00
compress time/Food 1362242616.6 ns (2785081.721249938) 1418301206.2 ns (12123495.950000048) 0.96
compress time/Food throughput 332718229 bytes 332718229 bytes 1
parquet_rs-zstd compress time/Food 1123988458.9 ns (1421954.4499999285) 1109482993.6 ns (1126088.1549999714) 1.01
parquet_rs-zstd compress time/Food throughput 332718229 bytes 332718229 bytes 1
decompress time/Food 178418315.08988097 ns (815796.0742128193) 182334756.50972223 ns (611796.5767569542) 0.98
decompress time/Food throughput 332718229 bytes 332718229 bytes 1
parquet_rs-zstd decompress time/Food 217377009.56666666 ns (469199.9933333248) 218510111.1 ns (395427.3354166597) 0.99
parquet_rs-zstd decompress time/Food throughput 332718229 bytes 332718229 bytes 1
vortex:parquet-zstd size/Food 1.27594400807971 ratio 1.2763820979031943 ratio 1.00
vortex:raw size/Food 0.13893878955456931 ratio 0.13898649358343393 ratio 1.00
vortex size/Food 46227468 bytes 46243340 bytes 1.00
compress time/HashTags 2862151910.4 ns (2595820.308749914) 2871625354.3 ns (4505561.451250076) 1.00
compress time/HashTags throughput 804495592 bytes 804495592 bytes 1
parquet_rs-zstd compress time/HashTags 2530710282.1 ns (1907594.9100000858) 2493443113.8 ns (3724863.971250057) 1.01
parquet_rs-zstd compress time/HashTags throughput 804495592 bytes 804495592 bytes 1
decompress time/HashTags 580692247.8 ns (2162342.1500000358) 580839751.2 ns (3493674.3474999666) 1.00
decompress time/HashTags throughput 804495592 bytes 804495592 bytes 1
parquet_rs-zstd decompress time/HashTags 764095601.8 ns (2698139.7575000525) 769704699.6 ns (6052454.925000012) 0.99
parquet_rs-zstd decompress time/HashTags throughput 804495592 bytes 804495592 bytes 1
vortex:parquet-zstd size/HashTags 1.6723313042885117 ratio 1.6722204697809582 ratio 1.00
vortex:raw size/HashTags 0.2784784630616099 ratio 0.27846000677651944 ratio 1.00
vortex size/HashTags 224034696 bytes 224019848 bytes 1.00
compress time/TPC-H l_comment chunked without fsst 4387359533.7 ns (11079425.331249714) 4399295314.5 ns (20324079.610000134) 1.00
compress time/TPC-H l_comment chunked without fsst throughput 249197090 bytes 249197090 bytes 1
parquet_rs-zstd compress time/TPC-H l_comment chunked without fsst 910439439.3 ns (1948368.3512499928) 911982655.9 ns (5191981.984999955) 1.00
parquet_rs-zstd compress time/TPC-H l_comment chunked without fsst throughput 249197090 bytes 249197090 bytes 1
decompress time/TPC-H l_comment chunked without fsst 215691321.73333335 ns (425363.606250003) 212065352.7 ns (753347.5337499976) 1.02
decompress time/TPC-H l_comment chunked without fsst throughput 249197090 bytes 249197090 bytes 1
parquet_rs-zstd decompress time/TPC-H l_comment chunked without fsst 249712543.25 ns (985445.1456250101) 248513324.70000005 ns (692517.8554166704) 1.00
parquet_rs-zstd decompress time/TPC-H l_comment chunked without fsst throughput 249197090 bytes 249197090 bytes 1
vortex:parquet-zstd size/TPC-H l_comment chunked without fsst 4.6096900085061705 ratio 4.609537186045604 ratio 1.00
vortex:raw size/TPC-H l_comment chunked without fsst 1.0531523622527053 ratio 1.0531690558665834 ratio 1.00
vortex size/TPC-H l_comment chunked without fsst 262442504 bytes 262446664 bytes 1.00
compress time/TPC-H l_comment chunked 1317330271.7 ns (1391043.6212500334) 1333481609.6 ns (1865651.3737499714) 0.99
compress time/TPC-H l_comment chunked throughput 249197090 bytes 249197090 bytes 1
parquet_rs-zstd compress time/TPC-H l_comment chunked 908382917.7 ns (2363152.311250031) 908074071.2 ns (3149992.9225000143) 1.00
parquet_rs-zstd compress time/TPC-H l_comment chunked throughput 249197090 bytes 249197090 bytes 1
decompress time/TPC-H l_comment chunked 153280743.61154762 ns (481828.6446428597) 155088209.9893254 ns (371215.5474047512) 0.99
decompress time/TPC-H l_comment chunked throughput 249197090 bytes 249197090 bytes 1
parquet_rs-zstd decompress time/TPC-H l_comment chunked 251236300.6 ns (558669.443749994) 249950106.73333335 ns (875178.6316666901) 1.01
parquet_rs-zstd decompress time/TPC-H l_comment chunked throughput 249197090 bytes 249197090 bytes 1
vortex:parquet-zstd size/TPC-H l_comment chunked 1.3522641469797685 ratio 1.3591660385005664 ratio 0.99
vortex:raw size/TPC-H l_comment chunked 0.3089448917722113 ratio 0.3105369488865219 ratio 0.99
vortex size/TPC-H l_comment chunked 76988168 bytes 77384904 bytes 0.99
compress time/TPC-H l_comment canonical 1322406722 ns (1068137.9687501192) 1341145376.05 ns (3394611.649999976) 0.99
compress time/TPC-H l_comment canonical throughput 249197106 bytes 249197106 bytes 1
parquet_rs-zstd compress time/TPC-H l_comment canonical 911378259.55 ns (1860989.4750000238) 907579165.1 ns (2102220.1999999285) 1.00
parquet_rs-zstd compress time/TPC-H l_comment canonical throughput 249197106 bytes 249197106 bytes 1
decompress time/TPC-H l_comment canonical 152718910.48434526 ns (367266.948482126) 154389045.62001982 ns (402580.30208332837) 0.99
decompress time/TPC-H l_comment canonical throughput 249197106 bytes 249197106 bytes 1
parquet_rs-zstd decompress time/TPC-H l_comment canonical 249264195.70646825 ns (325469.5102291703) 247579540.73156747 ns (279366.65624703467) 1.01
parquet_rs-zstd decompress time/TPC-H l_comment canonical throughput 249197106 bytes 249197106 bytes 1
vortex:parquet-zstd size/TPC-H l_comment canonical 1.3522721751830458 ratio 1.359157993684654 ratio 0.99
vortex:raw size/TPC-H l_comment canonical 0.3089448719360328 ratio 0.3105369289481235 ratio 0.99
vortex size/TPC-H l_comment canonical 76988168 bytes 77384904 bytes 0.99

This comment was automatically generated by workflow using github-action-benchmark.

Copy link
Member

@robert3005 robert3005 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's very verbose to call take, not sure how to make it nicer if you want default options

@gatesn gatesn merged commit 508db10 into develop Nov 17, 2024
12 checks passed
@gatesn gatesn deleted the ngates/unchecked-take branch November 17, 2024 18:59
@gatesn
Copy link
Contributor Author

gatesn commented Nov 17, 2024

I wanted to make sure the caller has to think about it! We could have done take_with_options but it would be too easy to forget to propagate them to your children IMO.

Comment on lines +8 to +10
pub struct TakeOptions {
pub skip_bounds_check: bool,
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

being able to toggle off bounds checking within a safe function feels kinda off...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed an improvement for a couple of benchmarks, so figured I'd add it. Initially it was local within Bool::take implementation, but seemed a shame not to generalize the logic.

What we could do, is to make TakeOptions public (in case we add any more), but then make the setter for skip_bounds_check non-public, meaning only the take entry-point function has the ability to set it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess the other problem here is it means we have to trust the stats from the file?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants