Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parquet: Fix performance regression in reader init #12305

Merged
merged 1 commit into from
Feb 19, 2025
Merged

Parquet: Fix performance regression in reader init #12305

merged 1 commit into from
Feb 19, 2025

Conversation

bryanck
Copy link
Contributor

@bryanck bryanck commented Feb 18, 2025 •
edited
Loading

Updating the Parquet libraries to 1.15 resulted in a performance regression when initializing a Parquet reader. In 1.15, when ParquetReadOptions.builder() is called with no parameter, a new HadoopParquetConfiguration is created. This ends up loading a Hadoop Configuration, which is an expensive operation.

This PR passes in an empty PlainParquetConfiguration when creating the builder to avoid loading the Hadoop configuration object.

@bryanck bryanck marked this pull request as ready for review February 18, 2025 06:33
Fokko
Fokko approved these changes Feb 18, 2025
View reviewed changes
Copy link
Contributor

@Fokko Fokko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, and also one fewer dependency on Hadoop 👍

@bryanck bryanck added this to the Iceberg 1.8.1 milestone Feb 18, 2025
singhpk234
singhpk234 approved these changes Feb 18, 2025
View reviewed changes
Copy link
Contributor

@singhpk234 singhpk234 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM as well ! Thanks @bryanck !

@nastra nastra merged commit c1d4182 into apache:main Feb 19, 2025
46 checks passed
nastra pushed a commit to nastra/iceberg that referenced this pull request Feb 19, 2025
@bryanck @nastra
Parquet: Fix performance regression in reader init (apache#12305)
8237cdf
@nastra nastra mentioned this pull request Feb 19, 2025
Merged
nastra added a commit that referenced this pull request Feb 19, 2025
@nastra @bryanck
Parquet: Fix performance regression in reader init (#12305) (#12329)
686bae5 
Co-authored-by: Bryan Keller <bryanck@gmail.com>
ankurbansal-tradedoubler added a commit to ankurbansal-tradedoubler/iceberg that referenced this pull request Feb 19, 2025
@ankurbansal-tradedoubler @danicafine
@manuzhang @ebyhr @dependabot @dramaticlly @nastra @tomtongue @rdblue @aihuaxu @Fokko @czy006 @bryanck @danielcweeks @pvary
update (#1)
2ffcb87 
* Site: Learn More to point to Spark QuickStart Doc (apache#12272)

* Build: Bump datamodel-code-generator from 0.27.2 to 0.28.1 (apache#12290)

* Spark 3.5: Fix job description of RewriteTablePathSparkAction (apache#12282)

* Build: Bump io.netty:netty-buffer from 4.1.117.Final to 4.1.118.Final (apache#12287)

Bumps [io.netty:netty-buffer](https://github.com/netty/netty) from 4.1.117.Final to 4.1.118.Final.
- [Commits](netty/netty@netty-4.1.117.Final...netty-4.1.118.Final)

---
updated-dependencies:
- dependency-name: io.netty:netty-buffer
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Build: Bump software.amazon.awssdk:bom from 2.30.16 to 2.30.21 (apache#12286)

Bumps software.amazon.awssdk:bom from 2.30.16 to 2.30.21.

---
updated-dependencies:
- dependency-name: software.amazon.awssdk:bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* OpenAPI: Add overwrite option when registering a table (apache#12239)

* OpenAPI: Add optional overwrite when registering table

* simplify to overwrite

* Add the article to the description

Co-authored-by: Eduard Tudenhoefner <etudenhoefner@gmail.com>

* Update generated python as well

Signed-off-by: Hongyue Zhang <steveiszhy@gmail.com>

* Fix import order

---------

Signed-off-by: Hongyue Zhang <steveiszhy@gmail.com>
Co-authored-by: Eduard Tudenhoefner <etudenhoefner@gmail.com>

* Build: Bump mkdocs-material from 9.6.3 to 9.6.4 (apache#12284)

Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.6.3 to 9.6.4.
- [Release notes](https://github.com/squidfunk/mkdocs-material/releases)
- [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG)
- [Commits](squidfunk/mkdocs-material@9.6.3...9.6.4)

---
updated-dependencies:
- dependency-name: mkdocs-material
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Core: Fix Enabling row-lineage during Create Table (apache#12307)

* API: Reject unknown type for required fields and validate defaults (apache#12302)

* API: Fix TestInclusiveMetricsEvaluator notStartsWith tests. (apache#12303)

* Core: Add variant type support to utils and visitors (apache#11831)

* Core: Fix CI: Update tests with UnknownType from required to optional (apache#12316)

* Docs: Refactor site navigation bar (apache#12289)

* Parquet: Implement Variant readers (apache#12139)

* Docs: Add rewrite_table_path Spark Procedure (apache#12115)

* Parquet: Fix errorprone warning (apache#12324)

* Docs: Add Apache Amoro docs (apache#11966)

* Parquet: Fix performance regression in reader init (apache#12305)

* Core: Fallback to GET requests for namespace/table/view exists checks (apache#12314)

Co-authored-by: Daniel Weeks <dweeks@apache.org>

* Docs: Fix refs in Apache Amoro docs (apache#12332)

* Revert "Core: Serialize `null` when there is no current snapshot (apache#11560)" (apache#12312)

This reverts commit bf8d25f.

* Parquet: Fix performance regression in reader init (apache#12305) (apache#12329)

Co-authored-by: Bryan Keller <bryanck@gmail.com>

* Checkstyle: Apply the same generic type naming rules to interfaces and classes (apache#12333)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Hongyue Zhang <steveiszhy@gmail.com>
Co-authored-by: Danica Fine <danica.fine@gmail.com>
Co-authored-by: Manu Zhang <OwenZhang1990@gmail.com>
Co-authored-by: Yuya Ebihara <ebyhry@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Hongyue/Steve Zhang <steveiszhy@gmail.com>
Co-authored-by: Eduard Tudenhoefner <etudenhoefner@gmail.com>
Co-authored-by: Tom Tanaka <43331405+tomtongue@users.noreply.github.com>
Co-authored-by: Ryan Blue <blue@apache.org>
Co-authored-by: Aihua Xu <aihuaxu@gmail.com>
Co-authored-by: Fokko Driesprong <fokko@apache.org>
Co-authored-by: ConradJam <jam.gzczy@gmail.com>
Co-authored-by: Bryan Keller <bryanck@gmail.com>
Co-authored-by: Daniel Weeks <dweeks@apache.org>
Co-authored-by: pvary <peter.vary.apache@gmail.com>
nastra added a commit that referenced this pull request Feb 19, 2025
@nastra @bryanck
Parquet: Fix performance regression in reader init (#12305) (#12338)
0bbf855 
Co-authored-by: Bryan Keller <bryanck@gmail.com>
nastra added a commit to nastra/iceberg that referenced this pull request Feb 26, 2025
@nastra @bryanck
Parquet: Fix performance regression in reader init (apache#12305) (ap…
d33e557 
…ache#12329)

Co-authored-by: Bryan Keller <bryanck@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nastra nastra nastra approved these changes

@Fokko Fokko Fokko approved these changes

@stevenzwu stevenzwu stevenzwu approved these changes

@singhpk234 singhpk234 singhpk234 approved these changes

Assignees
No one assigned
Labels
parquet
Projects
None yet
Milestone
Iceberg 1.8.1
Development

Successfully merging this pull request may close these issues.
5 participants
@bryanck @nastra @Fokko @stevenzwu @singhpk234