chore(deps): update parquet/arrow/arrow-csv from 56 to 57

[kevinjqliu]

Closes #197 #198 #199

[kevinjqliu]

looks like we need to refactor get_column_writers since its now deprecated

[clflushopt]

I am wondering what the impact is of moving to 57 for all three deps

[kevinjqliu]

I am wondering what the impact is of moving to 57 for all three deps

in terms of performance?

[alamb]

Thanks @kevinjqliu and @clflushopt

I ran some unscientific benchmarks on my laptop and indeed it seems like the upgrade makes writing 10% slower for some reason 🤔

Benchmark

rm -rf lineitem && time tpchgen-cli --scale-factor=100 --tables=lineitem --parts=10 --format=parquet

Release	Time
main	0m25.369s
main	0m25.729s
this PR	0m28.516s
this PR	0m28.682s

I'll do some profiling and see if I can see any reason

[alamb]

we just need to change this to follow the example here: https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/struct.ArrowColumnWriter.html

[kevinjqliu]

Not scientific either, this shows improvements for this PR vs last release (2.0.1)
only changes since 2.0.1 release are related to github actions, v2.0.1...main

➜ hyperfine --warmup 5 --runs 30 './target/release/tpchgen-cli -s 10'
Benchmark 1: ./target/release/tpchgen-cli -s 10
  Time (mean ± σ):     729.7 ms ±   7.6 ms    [User: 702.9 ms, System: 20.1 ms]
  Range (min … max):   714.5 ms … 748.4 ms    30 runs
 
➜ hyperfine --warmup 5 --runs 30 'uvx tpchgen-cli -s 10'
Benchmark 1: uvx tpchgen-cli -s 10
  Time (mean ± σ):     825.4 ms ±  12.3 ms    [User: 706.8 ms, System: 29.4 ms]
  Range (min … max):   812.7 ms … 873.6 ms    30 runs

[alamb]

Not scientific either, this shows improvements for this PR vs last release (2.0.1) only changes since 2.0.1 release are related to github actions, v2.0.1...main

➜ hyperfine --warmup 5 --runs 30 './target/release/tpchgen-cli -s 10'
Benchmark 1: ./target/release/tpchgen-cli -s 10
  Time (mean ± σ):     729.7 ms ±   7.6 ms    [User: 702.9 ms, System: 20.1 ms]
  Range (min … max):   714.5 ms … 748.4 ms    30 runs
 
➜ hyperfine --warmup 5 --runs 30 'uvx tpchgen-cli -s 10'
Benchmark 1: uvx tpchgen-cli -s 10
  Time (mean ± σ):     825.4 ms ±  12.3 ms    [User: 706.8 ms, System: 29.4 ms]
  Range (min … max):   812.7 ms … 873.6 ms    30 runs

I think by default tpchgen-cli makes TBL files (not parquet) so this command is likely not testing any changes related to arrow/parquet

[alamb]

I filed a ticket upstream to investigate and will post my findings there

• [Parquet] Writing in 57.0.0 seems 10% slower than 56.0.0 apache/arrow-rs#8783

[kevinjqliu]

I think by default tpchgen-cli makes TBL files (not parquet) so this command is likely not testing any changes related to arrow/parquet

oh yea, good point. I did it agains with the same options you used above. This PR is faster than v2.0.1

➜ hyperfine --warmup 5 --runs 30 './target/release/tpchgen-cli --scale-factor=100 --tables=lineitem --parts=10 --format=parquet'
Benchmark 1: ./target/release/tpchgen-cli --scale-factor=100 --tables=lineitem --parts=10 --format=parquet
  Time (mean ± σ):     784.5 ms ±   3.4 ms    [User: 757.3 ms, System: 20.9 ms]
  Range (min … max):   777.0 ms … 791.4 ms    30 runs

➜ hyperfine --warmup 5 --runs 30 'uvx tpchgen-cli --scale-factor=100 --tables=lineitem --parts=10 --format=parquet'
Benchmark 1: uvx tpchgen-cli --scale-factor=100 --tables=lineitem --parts=10 --format=parquet
  Time (mean ± σ):     869.3 ms ±   5.3 ms    [User: 753.4 ms, System: 29.8 ms]
  Range (min … max):   860.0 ms … 882.8 ms    30 runs

[alamb]

I think we have figured out what was going on

• [Parquet] Writing in 57.0.0 seems 10% slower than 56.0.0 apache/arrow-rs#8783

[clflushopt]

I wanted to confirm similar numbers, I spent a little while thinking I was doing something wrong 😮‍💨

[alamb]

I wanted to confirm similar numbers, I spent a little while thinking I was doing something wrong 😮‍💨

The good (amazing ❤️ ) news is that @etseidl has a fix already (will be released in 57.1.0):

• perf: Speed up Parquet file writing (10%, back to speed of 56) apache/arrow-rs#8786

I verified with the code in apache/arrow-rs#8786, this PR now has roughly similar performance to 56

It still seems a few percent slower (0m25.957s vs 0m25.378s) but I will file some issues upstream to further optimize things