[Parquet] ParquetMetadataReader decodes too much metadata under point-get scenerio

[mapleFU]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

When PageIndexPolicy is not Skip, the ParquetMetadataReader will read the whole page index and decoding them. The problem is that:

1. For large scan with all columns and all row-groups, this works well, decode metadata will be lightweight
2. For point-get with few columns, current it will read all column indexes and page index. Decoding them would be heavyweight sometimes

Describe the solution you'd like

A interface for row_group selection and only decoding them?

Describe alternatives you've considered

I don't know, maybe I can do io myself?

Additional context

No

[etseidl]

Thanks @mapleFU, you are correct. But please see #5853, and in particular #5855 and recent work to address this issue (#7582, #8643, #8713, #8714). The whole point of the thrift remodel was to address this 😄.

[mapleFU]

I've using a hacking way to solve this problem based on the current code:

1. ParquetMetadataReader call PageIndexPolicy::Skip
2. Construct a ParquetMetadata and do filtering with all row-groups, which filter out the row-groups I required
3. With the selected row-groups, I reconstruct a ParquetMetadataReader with existing metadata and Bytes buffer, and read only "selected row-groups" and "selected columns".
4. Decoding them. Offset-index of every column we required would be decoded, Column Index only need decode the "filter columns"

The way above still decode the row-group metadata, but it would be far more less than page indexes' metadata.

[mapleFU]

Basing on current code, partial decoding PageIndex is much more easier than decoding RowGroup. Maybe we can just think how to tell the reader that the PageIndex only need these row-group / columns

[etseidl]

Note that SerializedFileReader already does row group filtering

arrow-rs/parquet/src/file/serialized_reader.rs

Lines 200 to 216 in bac0cb5

	// Filter row groups based on the predicates
	for (i, rg_meta) in metadata_builder.take_row_groups().into_iter().enumerate() {
	let mut keep = true;
	for predicate in &mut predicates {
	if !predicate(&rg_meta, i) {
	keep = false;
	break;
	}
	}
	if keep {
	metadata_builder = metadata_builder.add_row_group(rg_meta);
	}
	}
	let mut metadata = metadata_builder.build();
	// If page indexes are desired, build them with the filtered set of row groups

.

This will only read the page indexes for the row groups that pass the filtering stage.

[etseidl]

Basing on current code, partial decoding PageIndex is much more easier than decoding RowGroup. Maybe we can just think how to tell the reader that the PageIndex only need these row-group / columns

Again, please take part in #8643. I just want to do this once 😅

[mapleFU]

Ah! Sorry for that, I'll close this and shift to #8643 .

[etseidl]

Ah! Sorry for that, I'll close this and shift to #8643 .

No worries, I just wanted you to take part since you have good ideas 😄