Refactor create_hashes to accept array references

[adriangb]

Background

This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in #17171.

A "target state" is tracked in #18393.
There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own:

• (This PR): Refactor create_hashes to accept array references #18448
• Refactor InListExpr to store arrays and support structs #18449 (depends on Refactor create_hashes to accept array references #18448)
• Refactor state management in HashJoinExec and use CASE expressions for more precise filters #18451

Changes in this PR

Change create_hashes and related functions to work with &dyn Array references instead of requiring ArrayRef (Arc-wrapped arrays). This avoids unnecessary Arc::clone() calls and enables calls that only have an &dyn Array to use the hashing utilities.

• Add create_hashes_from_arrays(&[&dyn Array]) function
• Refactor hash_dictionary, hash_list_array, hash_fixed_list_array to use references instead of cloning
• Extract hash_single_array() helper for common logic

[2010YOUY01]

What is this series of PRs implementing? Is it for query anti-pattern in #18393 (comment), stat pruning is not working, so we're pushing down build-side dynamic filter like key in [1,5,23] to probe side parquet scan, and use parquet builtin bloom filter to effectively skip prunable units?

By the way I suggest to open an umbrella issue to describe the high-level ideas, now the discussion seems to be scattered around, and the work is hard to follow.

[adriangb]

What is this series of PRs implementing? Is it for query anti-pattern in #18393 (comment), stat pruning is not working, so we're pushing down build-side dynamic filter like key in [1,5,23] to probe side parquet scan, and use parquet builtin bloom filter to effectively skip prunable units?

Yes that is precisely it. I'm not sure what you mean by query anti-pattern though.

I'll put a summary and links to the relevant PRs in #17171 (comment). That said #17171 is already a pretty busy issue so it's buried at the bottom... I'm not sure what more I can do about that.

[2010YOUY01]

What is this series of PRs implementing? Is it for query anti-pattern in #18393 (comment), stat pruning is not working, so we're pushing down build-side dynamic filter like key in [1,5,23] to probe side parquet scan, and use parquet builtin bloom filter to effectively skip prunable units?

Yes that is precisely it. I'm not sure what you mean by query anti-pattern though.

Maybe query anti-pattern is not accurate, but I'm just describing in that query, stat pruning is not working due to the given parquet file is not well clustered on the pruning column.

I'll put a summary and links to the relevant PRs in #17171 (comment). That said #17171 is already a pretty busy issue so it's buried at the bottom... I'm not sure what more I can do about that.

Perhaps edit the issue top description, or open a new one? 🤔

[adriangb]

Maybe query anti-pattern is not accurate, but I'm just describing in that query, stat pruning is not working due to the given parquet file is not well clustered on the pruning column.

Okay yes agreed. Anti-pattern was only confusing to me because I thought it might be suggesting that pattern of query is wrong / that the query itself is wrong and had me thinking maybe my example was a bad one 😔

I'll put a summary and links to the relevant PRs in #17171 (comment). That said #17171 is already a pretty busy issue so it's buried at the bottom... I'm not sure what more I can do about that.

Perhaps edit the issue top description, or open a new one? 🤔

Hmm there's others working on the same issue I don't want to make my work sounds like the only solution. I'll add a more extensive description and cross references to the PRs tomorrow.