Pass options to Parquet metadata readers

[etseidl]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

• Part of [EPIC] A collection of items to improve speed of parquet metadata encoding #5853

One of the goals of the Thrift remodel project (#5854) was to enable such things as selective decoding of parts of the Parquet metadata. The parsers are now in place to enable this, but was is lacking now is a way to communicate what bits of the metadata are required.

Describe the solution you'd like
Some mechanism to communicate to the metadata parsers what is needed. Options can include such things as:

• Skip some statistics fields in ColumnMetaData (Statistics, PageEncodingStatistics, SizeStatistics, etc).
• Parse page encoding statistics into some other form (boolean, bitmask) to support dictionary based pushdown.
• Column projections (i.e. skip decoding metadata/page indexes for columns that will not be read).
• Row group selection (only parse metadata for requested set of row groups).
• Only decode chunk statistics/column indexes for columns used in predicates.
• Only return schema.
• Skip schema and use a provided schema (perhaps from an earlier decode).
• Add an optional "skip index" that will enable random access to the metadata
• Perhaps move encryption parameters here as well.
• Others I haven't yet thought of.

Describe alternatives you've considered
These options could be added to current properties objects, but there doesn't seem to b a single place for all of these. For instance, SerializedFileReader takes a ReadOptions, that contains a ReaderProperties which is what is subsequently used by the SerialzedRowGroupReader and children. On the arrow side we instead use an ArrowReaderOptions. The ParquetMetaDataReader and ParquetMetaDataPushDecoder manage their own set of options. It would be nice to have a single place to set metadata parsing options and then pass that to the respective decoders.

Additional context

[alamb]

The ParquetMetaDataReader and ParquetMetaDataPushDecoder manage their own set of options.

I looked at the code and I think the only real options now are "should I read the page index":

• https://docs.rs/parquet/latest/src/parquet/file/metadata/reader.rs.html#66-75
• https://docs.rs/parquet/latest/src/parquet/file/metadata/push_decoder.rs.html#221-232

What if we made a new(!!!) options object like

struct ParquetMetadataOptions {
...
}

And then passed it to the ParquetMetadataReader (which now just wraps the MetadataPushDecoder)?

[alamb]

Another interesting API question: How do we represent the metadata for a subset of columns/row groups?

Right now ParquetMetaData has apis in terms of 'row group index' (and column index)

arrow-rs/parquet/src/file/metadata/mod.rs

Lines 243 to 247 in e9ea12b

	/// Returns row group metadata for `i`th position.
	/// Position should be less than number of row groups `num_row_groups`.
	pub fn row_group(&self, i: usize) -> &RowGroupMetaData {
	&self.row_groups[i]
	}

SO if we only got the RowGroupMetadata for 1 Row Group from a file with 10 Row Groups what would metadata.num_row_groups() return?

The same question applies to columns chunks too I think

[alamb]

Finally, another interesting question is if the ArrowReader should try and minimize the metadata decoding on its own.

For example, if the reader is asked to read only 3 columns, and no other instruction is given for metadata, should it only decode the metadata for those three columns?

I think the answer is yes....

[etseidl]

The ParquetMetaDataReader and ParquetMetaDataPushDecoder manage their own set of options.

I looked at the code and I think the only real options now are "should I read the page index":

* https://docs.rs/parquet/latest/src/parquet/file/metadata/reader.rs.html#66-75

* https://docs.rs/parquet/latest/src/parquet/file/metadata/push_decoder.rs.html#221-232

What if we made a new(!!!) options object like

struct ParquetMetadataOptions {
...
}

And then passed it to the ParquetMetadataReader (which now just wraps the MetadataPushDecoder)?

That's what I was thinking...but then who creates it and where? I'm starting to trace back from the metadata parsers to their calling sites to see what might make the most sense.

[etseidl]

Finally, another interesting question is if the ArrowReader should try and minimize the metadata decoding on its own.

For example, if the reader is asked to read only 3 columns, and no other instruction is given for metadata, should it only decode the metadata for those three columns?

I think the answer is yes....

I think so as well.

I need to look back at old discussions, but IIRC there was a suggestion to stash the footer bytes, and then materialize bits of metadata on demand. With an index this now becomes possible. That could solve the "how do we index this" question.

Edit: it was @XiangpengHao #5855 (comment). With the index as part of the footer, the penalty when wanting to read the entire file goes away.

[alamb]

Finally, another interesting question is if the ArrowReader should try and minimize the metadata decoding on its own.
For example, if the reader is asked to read only 3 columns, and no other instruction is given for metadata, should it only decode the metadata for those three columns?
I think the answer is yes....

I think so as well.

I need to look back at old discussions, but IIRC there was a suggestion to stash the footer bytes, and then materialize bits of metadata on demand. With an index this now becomes possible. That could solve the "how do we index this" question.

Edit: it was @XiangpengHao #5855 (comment). With the index as part of the footer, the penalty when wanting to read the entire file goes away.

I think the code in arrow-rs / parquet-rs should just do the best with what it has and leave additional caching / optimization to other layers.

For example, in DataFusion, now the code already caches the entire ParquetMetaData (including column index) and passes it into the arrow-rs code for all columns in many cases, so adding additional caching in the parquet reader itself seems unecessary.

What I think would help is APIs for progressively reading / populating the metadata (e.g. initially only read 5 columns, but then be able to incrementally parse / produce the remaining columns after) -- this maybe is APIs on ParquetMetaData to add new columns, / row groups

[etseidl]

I plan to start on this with a single concrete use (like read only schema/reuse schema across files) just to get the plumbing in place. Then we can start adding more use cases, such as selective decoding of page indexes, then selective decoding of column chunk metadata. I imagine this will be part of the public API, so I don't want to add a bunch of speculative features up front, only to finding we need to change them later and having to make breaking changes.

[mapleFU]

As @etseidl tells me, the code below ( which is under parquet and not under arrow module ) would read the required row-group.

arrow-rs/parquet/src/file/serialized_reader.rs

Lines 200 to 216 in bac0cb5

	// Filter row groups based on the predicates
	for (i, rg_meta) in metadata_builder.take_row_groups().into_iter().enumerate() {
	let mut keep = true;
	for predicate in &mut predicates {
	if !predicate(&rg_meta, i) {
	keep = false;
	break;
	}
	}
	if keep {
	metadata_builder = metadata_builder.add_row_group(rg_meta);
	}
	}
	let mut metadata = metadata_builder.build();
	// If page indexes are desired, build them with the filtered set of row groups

What I think would help is APIs for progressively reading / populating the metadata (e.g. initially only read 5 columns, but then be able to incrementally parse / produce the remaining columns after) -- this maybe is APIs on ParquetMetaData to add new columns, / row groups

I think there would be a interface or stage between load footer statics ( Like the code above, perhaps between DecodeState::ReadingPageIndex and DecodeState::Finished). Maybe add new statistics here is a good way for page index.

Besides, this might work like ArrowPredicate in arrow module: statistics/column index needs the ArrowPredicate::projection columns, and other required columns should read all offset index. I'm new here and I don't know would ArrowPredicate also be used here?

[alamb]

I think there would be a interface or stage between load footer statics ( Like the code above, perhaps between DecodeState::ReadingPageIndex and DecodeState::Finished). Maybe add new statistics here is a good way for page index.

I don't quite understand this proposal