Speedup statistics_from_parquet_metadata

[Dandandan]

Which issue does this PR close?

• Closes Speedup statistics_from_parquet_metadata (DataFusion side) #20005

PR:

SELECT COUNT(*) FROM hits WHERE "AdvEngineID" <> 0;

Query 1 iteration 0 took 30.5 ms and returned 1 rows
Query 1 avg time: 30.48 ms

Main:

SELECT COUNT(*) FROM hits WHERE "AdvEngineID" <> 0;

Query 1 iteration 0 took 39.6 ms and returned 1 rows
Query 1 avg time: 39.61 ms

Rationale for this change

Improving cold starts.

What changes are included in this PR?

Are these changes tested?

Existing tests

Are there any user-facing changes?

No

[alamb]

Thanks @Dandandan -- this looks good to me

It also makes me wonder if there is more performance to be had in our statistics management -- for example, I notice that there are multiple call sites that convert the max values (when the table is first created and then during query to try pruning)

https://github.com/search?q=repo%3Aapache%2Fdatafusion%20row_group_maxes&type=code

[alamb]

As an aside, there is the type if pattern where I would really like an API that can reuse the allocation

• [Arrow] Add bitwise operations BooleanArray that potentially reuse the underlying allocation arrow-rs#8809

This could easily reuse the allocation of eq_mask

(not tis PR)

[alamb]

It does seem like this fast path will be hit often (as it is when all stats are exact, a common case).

[Dandandan]

Yes!

[Dandandan]

🚀

[Dandandan]

Thanks @Dandandan -- this looks good to me

It also makes me wonder if there is more performance to be had in our statistics management -- for example, I notice that there are multiple call sites that convert the max values (when the table is first created and then during query to try pruning)

https://github.com/search?q=repo%3Aapache%2Fdatafusion%20row_group_maxes&type=code

I think there is also a lot of overhead in creating Arrays, accumulators, etc. (columnar datastructures) for essentially only a bunch of values...