Parquet: Support reading Parquet metadata via suffix range requests #7334

kylebarron · 2025-03-26T16:47:26Z

Which issue does this PR close?

Closes Read Parquet metadata via suffix requests #5979

Rationale for this change

Avoid needing to know file size in advance to read Parquet files.

What changes are included in this PR?

Add MetadataSuffixFetch trait which expands on MetadataFetch to additionally support reading suffix range requests.
Breaking: Change ParquetObjectReader::new to a signature of new(store: Arc<dyn ObjectStore>, path: Path, file_size: Option<usize>), so that file_size is no longer required.
Implement MetadataSuffixFetch for ParquetObjectReader, using ObjectStore::get_opts.
Always prefer reading metadata via bounded range requests if the file size is provided, and only fall back to suffix range requests if the file size is None.
Added simple test without file length

Are there any user-facing changes?

Yes, breaking change to the signature of ParquetObjectReader::new. I'd like to get this in to #7084.

Supersedes and closes #6157.

parquet/src/file/metadata/reader.rs

etseidl

Looking pretty good to me, just a few comments so far. I want to do a second pass later today or tomorrow.

Thanks!

parquet/src/file/metadata/reader.rs

alamb · 2025-03-28T00:19:31Z

I hope to review this tomorrow

alamb

Thank you so much @kylebarron -- this is a great improvement

I think the only thing that it needs is a test of some sort that shows the suffix was requested (to verify that we are minimizing the number of requests)

I also have some API suggestions and code refactoring ideas but I don't think they are needed.

All in all this is great. Thank you

parquet/src/file/metadata/reader.rs

alamb · 2025-03-28T15:42:20Z

parquet/src/file/metadata/reader.rs

+        prefetch: usize,
+    ) -> Result<(ParquetMetaData, Option<(usize, Bytes)>)> {
+        let suffix = fetch.fetch_suffix(prefetch.max(FOOTER_SIZE)).await?;
+        let suffix_len = suffix.len();


The code from here on down seems like it is copy/pasted from load_metadata -- could you please try and avoid duplication if possible by refactoring the common code into a shared method?

It's mostly copy/pasted but with a few differences between each

Whether to call fetch or fetch_suffix

arithmetic over byte ranges to read

There might be a way to DRY but I don't see anything super obvious other than maybe these four lines

let mut footer = [0; FOOTER_SIZE]; footer.copy_from_slice(&suffix[suffix_len - FOOTER_SIZE..suffix_len]); let footer = Self::decode_footer_tail(&footer)?; let length = footer.metadata_length();

parquet/src/arrow/async_reader/store.rs

kylebarron · 2025-03-28T17:35:29Z

I think the only thing that it needs is a test of some sort that shows the suffix was requested (to verify that we are minimizing the number of requests)

Do you have a suggestion of how to do this?

alamb · 2025-03-28T17:37:46Z

I think the only thing that it needs is a test of some sort that shows the suffix was requested (to verify that we are minimizing the number of requests)

Do you have a suggestion of how to do this?

Perhaps we can implement an object_store wrapper that wraps a InMemory store but records the requests it gets 🤔

We have used that pattern in the past to good effect elsewhere I think

kylebarron · 2025-03-28T18:06:37Z

@alamb what do you think about c5e4226 (#7334)?

alamb

Thank you @kylebarron -- this looks great to me. Thank you for the tests

alamb · 2025-03-28T18:20:51Z

parquet/src/file/metadata/reader.rs

+        assert_eq!(actual.file_metadata().schema(), expected);
+        assert_eq!(fetch_count.load(Ordering::SeqCst), 0);
+        assert_eq!(suffix_fetch_count.load(Ordering::SeqCst), 2);
+    }


Could you also please add a test case for passing in the metadata size hint? And the expected number of suffix fetches is 1?

alamb · 2025-03-28T20:35:38Z

You are the best. Thanks again for this improvement and the tests @kylebarron

alamb · 2025-03-31T14:11:47Z

Thanks again @kylebarron !

resolves #968 (I hope) ## What changes are proposed in this pull request? TLDR: Fixes kernel (default-engine) error when trying to read on azure. `arrow-55` introduced [support for reading parquet metadata via suffix range requests](apache/arrow-rs#7334) but since azure [doesn't support suffix range requests](https://github.com/apache/arrow-rs-object-store/blob/9c43521d3c8e86fdfc4df788a5a63e45e23093b7/src/azure/client.rs#L912-L918) whenever anyone tried reading an azure table with default-engine based on arrow-55 it [blew up](#968). This PR mitigates by re-adding the original `head` request so that we can pass the `file_size` to the `ParquetObjectReader::with_file_size` which states that when this API is used, the reader will ensure only bounded range requests are used: ```rust /// Provide the byte size of this file. /// /// If provided, the file size will ensure that only bounded range requests are used. If file /// size is not provided, the reader will use suffix range requests to fetch the metadata. /// /// Providing this size up front is an important optimization to avoid extra calls when the /// underlying store does not support suffix range requests. /// /// The file size can be obtained using [`ObjectStore::list`] or [`ObjectStore::head`]. pub fn with_file_size(self, file_size: usize) -> Self { ... } ``` ## How was this change tested? tested manually.. painfully manually. spun up a public azure container thing and pointed our example code at it. was able to repro the following error without my changes and observe it going away after changes here applied. ``` Parquet( External( NotSupported { source: "Azure does not support suffix range requests", }, ), ) ``` obviously highlights our test gap. working on setting up some environments to test against S3/ADLS/GCS/etc.

Support reading Parquet metadata via suffix requests

b2110d1

github-actions bot added the parquet Changes to the parquet crate label Mar 26, 2025

kylebarron commented Mar 26, 2025

View reviewed changes

parquet/src/file/metadata/reader.rs Outdated Show resolved Hide resolved

This was referenced Mar 26, 2025

Parquet/async: Default to suffix requests on supporting readers/object stores #6157

Closed

Release arrow-rs / parquet major version 55.0.0 (Apr 2025) #7084

Closed

kylebarron added 2 commits March 26, 2025 12:54

Update doctest

7930451

Add simple test without file length provided

133dc2d

etseidl reviewed Mar 26, 2025

View reviewed changes

parquet/src/file/metadata/reader.rs Outdated Show resolved Hide resolved

parquet/src/file/metadata/reader.rs Outdated Show resolved Hide resolved

parquet/src/file/metadata/reader.rs Show resolved Hide resolved

etseidl added api-change Changes to the arrow API next-major-release the PR has API changes and it waiting on the next major version labels Mar 26, 2025

address comments

3d96024

alamb reviewed Mar 28, 2025

View reviewed changes

kylebarron added 4 commits March 28, 2025 11:54

Switch file_size optional arg to with_file_size method

6b94524

Move prefetch lookup into load_metadat

c2e2fd6

Remove extra call to max

e3cbcda

fix doctest

057b75a

Add test that suffix requests are made

c5e4226

alamb approved these changes Mar 28, 2025

View reviewed changes

more tests

7a3e31f

alamb merged commit a3598a9 into apache:main Mar 31, 2025
17 checks passed

This was referenced Mar 31, 2025

Remove AsyncFileReader::get_metadata_with_options, add options to AsyncFileReader::get_metadata #7342

Merged

ArrowReaderMetadata API makes it too easy to (accidentally) make an additional object store request #6476

Closed

alamb mentioned this pull request Apr 11, 2025

Upgrade to arrow/parquet 55, and object_store to 0.12.0 and pyo3 to 0.24.0 apache/datafusion#15466

Merged

alamb mentioned this pull request Apr 30, 2025

feat: upgrade to DataFusion 47.0.0 delta-io/delta-rs#3378

Merged

zachschuermann mentioned this pull request Jun 5, 2025

fix: remove azure suffix range request delta-io/delta-kernel-rs#1006

Merged

etseidl mentioned this pull request Nov 25, 2024

Deprecate and replace old Parquet metadata parsing functions #6447

Closed

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Parquet: Support reading Parquet metadata via suffix range requests #7334

Parquet: Support reading Parquet metadata via suffix range requests #7334

kylebarron commented Mar 26, 2025 •

edited by alamb

Loading

Uh oh!

Uh oh!

etseidl left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alamb commented Mar 28, 2025

Uh oh!

alamb left a comment

Uh oh!

Uh oh!

alamb Mar 28, 2025

Uh oh!

kylebarron Mar 28, 2025

Uh oh!

Uh oh!

kylebarron commented Mar 28, 2025

Uh oh!

alamb commented Mar 28, 2025

Uh oh!

kylebarron commented Mar 28, 2025 •

edited

Loading

Uh oh!

alamb left a comment

Uh oh!

alamb Mar 28, 2025

Uh oh!

alamb commented Mar 28, 2025

Uh oh!

Uh oh!

alamb commented Mar 31, 2025

Uh oh!

Uh oh!

Parquet: Support reading Parquet metadata via suffix range requests #7334

Parquet: Support reading Parquet metadata via suffix range requests #7334

Conversation

kylebarron commented Mar 26, 2025 • edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Uh oh!

Uh oh!

etseidl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alamb commented Mar 28, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alamb Mar 28, 2025

Choose a reason for hiding this comment

Uh oh!

kylebarron Mar 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kylebarron commented Mar 28, 2025

Uh oh!

alamb commented Mar 28, 2025

Uh oh!

kylebarron commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Mar 28, 2025

Choose a reason for hiding this comment

Uh oh!

alamb commented Mar 28, 2025

Uh oh!

Uh oh!

alamb commented Mar 31, 2025

Uh oh!

Uh oh!

kylebarron commented Mar 26, 2025 •

edited by alamb

Loading

kylebarron commented Mar 28, 2025 •

edited

Loading