Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for versioning metastore API #15044

Merged
merged 4 commits into from
Sep 18, 2020
Merged

Add support for versioning metastore API #15044

merged 4 commits into from
Sep 18, 2020

Conversation

NikhilCollooru
Copy link
Contributor

@NikhilCollooru NikhilCollooru commented Aug 17, 2020 •
edited
Loading 

== RELEASE NOTES ==

General Changes
* Add support for partition versioning. This can be enabled with ``hive.partition-versioning-enabled`` configuration property
* Add support for caching HiveMetastore calls at a more granular level . Supported levels:  'PARTITION' and 'ALL'(default). This can be set with ``hive.metastore-cache-scope`` configuration property.

depended by https://github.com/facebookexternal/presto-facebook/pull/1127

shixuan-fan
shixuan-fan reviewed Aug 19, 2020
View reviewed changes
Copy link
Contributor

@shixuan-fan shixuan-fan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm debating with myself whether we should carry this version information over and let getPartitionsByNames handle version check. But this might need more changes and break more interfaces so I'm not sure.

Also, currently a partition drop would invalidate all partitions for one table (See invalidatePartitionCache). Maybe we should change this as well.

...to-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/CachingHiveMetastore.java Outdated Show resolved Hide resolved
...to-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/thrift/HiveMetastore.java Show resolved Hide resolved
...to-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/CachingHiveMetastore.java Outdated Show resolved Hide resolved
@NikhilCollooru NikhilCollooru force-pushed the versioning branch 2 times, most recently from e031a29 to b44ed03 Compare September 9, 2020 07:49
shixuan-fan
shixuan-fan reviewed Sep 11, 2020
View reviewed changes
Copy link
Contributor

@shixuan-fan shixuan-fan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Add optional version parameter to metastore Partition"

...o-hive-metastore/src/test/java/com/facebook/presto/hive/metastore/TestHiveMetastoreUtil.java Outdated Show resolved Hide resolved
presto-hive-metastore/src/main/java/com/facebook/presto/hive/PartitionExtraInfoFetcher.java Outdated Show resolved Hide resolved
presto-hive-metastore/src/main/java/com/facebook/presto/hive/PartitionExtraInfoFetcher.java Outdated Show resolved Hide resolved
shixuan-fan
shixuan-fan reviewed Sep 11, 2020
View reviewed changes
Copy link
Contributor

@shixuan-fan shixuan-fan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Add config to enable Partition versioning"

presto-hive-metastore/src/main/java/com/facebook/presto/hive/MetastoreClientConfig.java Outdated Show resolved Hide resolved
shixuan-fan
shixuan-fan reviewed Sep 11, 2020
View reviewed changes
Copy link
Contributor

@shixuan-fan shixuan-fan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Add new getPartitionNamesWithVersionByFilter metastore api"

...to-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/CachingHiveMetastore.java Outdated Show resolved Hide resolved
...to-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/CachingHiveMetastore.java Outdated Show resolved Hide resolved
...to-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/CachingHiveMetastore.java Outdated Show resolved Hide resolved
...to-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/CachingHiveMetastore.java Outdated Show resolved Hide resolved
...to-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/CachingHiveMetastore.java Outdated Show resolved Hide resolved
...to-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/CachingHiveMetastore.java Outdated Show resolved Hide resolved
...to-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/CachingHiveMetastore.java Outdated Show resolved Hide resolved
...to-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/CachingHiveMetastore.java Outdated Show resolved Hide resolved
...ive-metastore/src/main/java/com/facebook/presto/hive/metastore/PartitionNameWithVersion.java Outdated Show resolved Hide resolved
...to-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/CachingHiveMetastore.java Outdated Show resolved Hide resolved
shixuan-fan
shixuan-fan reviewed Sep 11, 2020
View reviewed changes
Copy link
Contributor

@shixuan-fan shixuan-fan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Add CachingHiveMetastoreConfig to manage metastore cache parameters"

...to-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/CachingHiveMetastore.java Outdated Show resolved Hide resolved
...e-metastore/src/main/java/com/facebook/presto/hive/metastore/CachingHiveMetastoreConfig.java Outdated Show resolved Hide resolved
shixuan-fan
shixuan-fan requested changes Sep 11, 2020
View reviewed changes
Copy link
Contributor

@shixuan-fan shixuan-fan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good. There are some nitty things need to be fixed

@NikhilCollooru NikhilCollooru force-pushed the versioning branch 3 times, most recently from c3ca890 to 90a8347 Compare September 11, 2020 22:49
@NikhilCollooru NikhilCollooru marked this pull request as ready for review September 11, 2020 23:38
shixuan-fan
shixuan-fan reviewed Sep 15, 2020
View reviewed changes
Copy link
Contributor

@shixuan-fan shixuan-fan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Add optional version parameter to metastore Partition" Small nits on renaming leftover

...metastore/src/main/java/com/facebook/presto/hive/metastore/thrift/BridgingHiveMetastore.java Outdated Show resolved Hide resolved
...e-metastore/src/main/java/com/facebook/presto/hive/metastore/thrift/ThriftMetastoreUtil.java Outdated Show resolved Hide resolved
...ive-metastore/src/test/java/com/facebook/presto/hive/metastore/TestCachingHiveMetastore.java Outdated Show resolved Hide resolved
shixuan-fan
shixuan-fan reviewed Sep 15, 2020
View reviewed changes
Copy link
Contributor

@shixuan-fan shixuan-fan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Add new getPartitionNamesWithVersionByFilter metastore api"

...to-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/CachingHiveMetastore.java Outdated Show resolved Hide resolved
...to-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/CachingHiveMetastore.java Outdated Show resolved Hide resolved
...to-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/CachingHiveMetastore.java Outdated Show resolved Hide resolved
...o-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/ExtendedHiveMetastore.java Outdated Show resolved Hide resolved
@NikhilCollooru NikhilCollooru force-pushed the versioning branch 2 times, most recently from 8a94c2d to 4bfaf38 Compare September 16, 2020 03:06
shixuan-fan
shixuan-fan approved these changes Sep 16, 2020
View reviewed changes
Copy link
Contributor

@shixuan-fan shixuan-fan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also add test plan and release note?

...to-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/CachingHiveMetastore.java Outdated Show resolved Hide resolved
...to-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/CachingHiveMetastore.java Outdated Show resolved Hide resolved
...to-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/CachingHiveMetastore.java Show resolved Hide resolved
presto-hive-metastore/src/main/java/com/facebook/presto/hive/PartitionVersionFetcher.java Show resolved Hide resolved
@jainxrohit jainxrohit self-requested a review September 17, 2020 19:24
@NikhilCollooru
Copy link
Contributor Author

@shixuan-fan I added a new commit to add the units tests.
I am not sure why the Facebook build is failing for this PR even after pointing it to the correct presto-facebook PR.

@shixuan-fan
Copy link
Contributor

shixuan-fan commented Sep 17, 2020 •
edited
Loading

I'm not sure why either. But I saw the internal PR build succeeded so we should be good. I clicked retry just in case.

@shixuan-fan shixuan-fan merged commit 4b8021f into prestodb:master Sep 18, 2020
@NikhilCollooru NikhilCollooru deleted the versioning branch September 18, 2020 19:09
This was referenced Oct 6, 2020
Merged
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shixuan-fan shixuan-fan shixuan-fan approved these changes

@jainxrohit jainxrohit Awaiting requested review from jainxrohit

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@NikhilCollooru @shixuan-fan @jainxrohit