Orc performance issue

[richox]

Describe the bug
we found orc scan has poor perfomance while running tpcds benchmark:

the same scan operator is times slower than parquet (from tpcds q3).

To Reproduce
Steps to reproduce the behavior:

1. generate parquet and orc datasets using /tpcds/datagen.
2. run bechmarks on both datasets using /tpcds/benchmark-runner.
3. compare the performance of NativeParquetScan and NativeOrcScan.

Expected behavior
orc should have the similar performance comparing to parquet.

Screenshots

Edit
the main reason is that orc-rust reads all data without column pruning and predicate filtering, after applying column pruning with datafusion-contrib/datafusion-orc#133 , the performance will be much better:

currently orc is still 20%~30% slower than parquet, which maybe related to unsupported predicate filtering.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.