Add statistics to ParquetExec for *files* pruned

[alamb]

Is your feature request related to a problem or challenge?

• This is a follow on to the feature added by @adriangb in Add late pruning of Parquet files based on file level statistics #16014

@adriangb added the great feature that can prune entire files while opening many parquet files

The current statistics for DataSourceExec have information on how many row groups were pruned, it would also be great to add statistics on how many FILES were pruned by this new code

For example, with clickbench Q24 here is an excerpt from the file

EXPLAIN ANALYZE SELECT "SearchPhrase" FROM hits WHERE "SearchPhrase" <> '' ORDER BY "SearchPhrase" LIMIT 10;

|                   |         DataSourceExec:...
 pushdown_rows_pruned=0, row_groups_matched_bloom_filter=0, row_groups_matched_statistics=325, row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0

Describe the solution you'd like

I would like some new statistics that record:

• files_pruned: total files that were pruned by filters during open

It is important to make sure the docs explain the metric only describes files pruned after the plan starts (not files that are pruned during planning)

Describe alternatives you've considered

1. Add a field to ParquetFileMetrics: https://github.com/apache/datafusion/blob/6d5e00ad3f8e53f7252cb1d3c72a6c7f28c1aed6/datafusion/datasource-parquet/src/metrics.rs#L29-L28
2. Thread that through to the opener in datafusion/datasource-parquet/src/opener.rs so when files are pruned we can see that in the metrics

Additional context

No response

[adriangb]

Doesn't

datafusion/datafusion/datasource-parquet/src/metrics.rs

Lines 30 to 33 in 4dd6923

	/// Number of files pruned by partition of file level statistics
	/// This often happens at planning time but may happen at execution time
	/// if dynamic filters (e.g. from a join) result in additional pruning.
	pub files_pruned_statistics: Count,

cover that?

[alamb]

cover that?

Yes 🤦

For some reason it doesn't show up for me in the explain analyze I have: q25-analyze-topk-dynamic-filter.txt

Which I made with this command from the #15770 branch

$ cat q25-analyze.sql
EXPLAIN ANALYZE SELECT "SearchPhrase" FROM hits WHERE "SearchPhrase" <> '' ORDER BY "SearchPhrase" LIMIT 10;
./datafusion-cli-topk-dynamic-filters -f q25-analyze.sql  > q25-analyze-topk-dynamic-filter.txt

But the query clearly got faster, so I would expect it to be present 🤔

[adriangb]

cover that?

Yes 🤦

For some reason it doesn't show up for me in the explain analyze I have: q25-analyze-topk-dynamic-filter.txt

Which I made with this command from the #15770 branch

$ cat q25-analyze.sql
EXPLAIN ANALYZE SELECT "SearchPhrase" FROM hits WHERE "SearchPhrase" <> '' ORDER BY "SearchPhrase" LIMIT 10;
./datafusion-cli-topk-dynamic-filters -f q25-analyze.sql > q25-analyze-topk-dynamic-filter.txt
But the query clearly got faster, so I would expect it to be present 🤔

Hmm maybe we aren't including that statistic in the output?

[alamb]

Hmm maybe we aren't including that statistic in the output?

I think everything that is non zero is included. I'll have to look into it some more

[adriangb]

Could it be that in that test we don't have file statistics (datafusion.execution.collect_statistics = false) -> the pruning is happening at the row group level?

[alamb]

Could it be that in that test we don't have file statistics (datafusion.execution.collect_statistics = false) -> the pruning is happening at the row group level?

set datafusion.execution.collect_statistics = true;
EXPLAIN ANALYZE SELECT "SearchPhrase" FROM hits WHERE "SearchPhrase" <> '' ORDER BY "SearchPhrase" LIMIT 10;

And that still doesn't seem to help

Let's wait for #15770 to merge and then I can dig into this more

[alamb]

(basically I want to be able to see from statistics when the dynamic filters are helping / not helping)