Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding presto-pinot-driver module #7384

Merged
merged 3 commits into from
Sep 9, 2021

Conversation

xiangfu0
Copy link
Contributor

@xiangfu0 xiangfu0 commented Sep 1, 2021

Description

Move presto-pinot-driver module from presto codebase (https://github.com/prestodb/presto-pinot-driver)

Presto repo is still on java 8, so we need to build this module with jdk8:

  1. Add a new profile to set jdk.version for all corresponding modules
  2. Add grpc client configs for presto
  3. Add helper functions for presto side to avoid dependency checks from class import

How to build:
So when we want to build presto-pinot-driver module, we need to first install two JDKs, e.g. JDK 8 and JDK 11 corresponding modules with JDK 8 lib.

Add profile presto-driver:

mvn clean install -DskipTests -Ppresto-driver -T 1C -pl ':presto-pinot-driver' -am

Upgrade Notes

Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)

  • Yes (Please label as backward-incompat, and complete the section below on Release Notes)

Does this PR fix a zero-downtime upgrade introduced earlier?

  • Yes (Please label this as backward-incompat, and complete the section below on Release Notes)

Does this PR otherwise need attention when creating release notes? Things to consider:

  • New configuration options
  • Deprecation of configurations
  • Signature changes to public methods/interfaces
  • New plugins added or old plugins removed
  • Yes (Please label this PR as release-notes and complete the section on Release Notes)

Release Notes

Documentation

@codecov-commenter
Copy link

codecov-commenter commented Sep 1, 2021

Codecov Report

Merging #7384 (257f35e) into master (c27ae81) will decrease coverage by 6.89%.
The diff coverage is 0.00%.

❗ Current head 257f35e differs from pull request most recent head edc0dff. Consider uploading reports for the commit edc0dff to get more accurate results
Impacted file tree graph

@@             Coverage Diff              @@
##             master    #7384      +/-   ##
============================================
- Coverage     71.89%   64.99%   -6.90%     
+ Complexity     3348     3345       -3     
============================================
  Files          1517     1471      -46     
  Lines         75052    73140    -1912     
  Branches      10936    10731     -205     
============================================
- Hits          53956    47536    -6420     
- Misses        17475    22226    +4751     
+ Partials       3621     3378     -243     
Flag Coverage Δ
integration1 ?
integration2 ?
unittests1 69.68% <0.00%> (-0.07%) ⬇️
unittests2 14.52% <0.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...pache/pinot/common/utils/grpc/GrpcQueryClient.java 0.00% <0.00%> (-100.00%) ⬇️
...a/org/apache/pinot/common/metrics/MinionMeter.java 0.00% <0.00%> (-100.00%) ⬇️
...g/apache/pinot/common/metrics/ControllerMeter.java 0.00% <0.00%> (-100.00%) ⬇️
.../apache/pinot/common/metrics/BrokerQueryPhase.java 0.00% <0.00%> (-100.00%) ⬇️
.../apache/pinot/common/metrics/MinionQueryPhase.java 0.00% <0.00%> (-100.00%) ⬇️
...he/pinot/common/messages/SegmentReloadMessage.java 0.00% <0.00%> (-100.00%) ⬇️
...pinot/core/data/manager/realtime/TimerService.java 0.00% <0.00%> (-100.00%) ⬇️
...pinot/minion/exception/TaskCancelledException.java 0.00% <0.00%> (-100.00%) ⬇️
...t/core/startree/plan/StarTreeDocIdSetPlanNode.java 0.00% <0.00%> (-100.00%) ⬇️
...ot/common/restlet/resources/TableMetadataInfo.java 0.00% <0.00%> (-100.00%) ⬇️
... and 353 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c27ae81...edc0dff. Read the comment docs.

@Jackie-Jiang
Copy link
Contributor

Can you please elaborate more on why we need to replicate some of the classes in this new module? And how is the module going to be released?

@xiangfu0 xiangfu0 marked this pull request as draft September 1, 2021 20:00
@xiangfu0 xiangfu0 force-pushed the adding-presto-pinot-driver-module branch 4 times, most recently from 40ae942 to 1132150 Compare September 8, 2021 08:37
@xiangfu0 xiangfu0 marked this pull request as ready for review September 8, 2021 08:37
@xiangfu0 xiangfu0 force-pushed the adding-presto-pinot-driver-module branch 11 times, most recently from a35b7ed to b6bd280 Compare September 8, 2021 09:21
@xiangfu0
Copy link
Contributor Author

xiangfu0 commented Sep 8, 2021

Can you please elaborate more on why we need to replicate some of the classes in this new module? And how is the module going to be released?

removed duplicated classes.

For releases, need to update the release doc of using the maven toolchain plugin

@yupeng9
Copy link
Contributor

yupeng9 commented Sep 8, 2021

Can you explain the purpose of this change?

@xiangfu0
Copy link
Contributor Author

xiangfu0 commented Sep 8, 2021

Can you explain the purpose of this change?

Can you explain the purpose of this change?

presto is on jdk8 and pinot is on jdk 11, so we need to recompile all pinot dependencies jars for presto usage.

@yupeng9
Copy link
Contributor

yupeng9 commented Sep 8, 2021

k. but then why there are other changes than the pom changes?

@xiangfu0
Copy link
Contributor Author

xiangfu0 commented Sep 8, 2021

k. but then why there are other changes than the pom changes?

After chatted with Prestodb community, I think it's better to move presto-pinot-driver (https://github.com/prestodb/presto-pinot-driver) module in pinot codebase, then refer to this module in presto plugin directly.

Presto side has extra configs for grpc client and we expose the scatter-gather client.
Other changes are due to some presto side integrations/checks.

@xiangfu0 xiangfu0 force-pushed the adding-presto-pinot-driver-module branch from 67ea73f to 87f7ecd Compare September 9, 2021 06:44
@xiangfu0 xiangfu0 force-pushed the adding-presto-pinot-driver-module branch from 87f7ecd to 529c819 Compare September 9, 2021 07:31
@xiangfu0 xiangfu0 force-pushed the adding-presto-pinot-driver-module branch from 529c819 to ca5e677 Compare September 9, 2021 19:12
@xiangfu0 xiangfu0 force-pushed the adding-presto-pinot-driver-module branch from ca5e677 to edc0dff Compare September 9, 2021 21:00
@xiangfu0 xiangfu0 requested a review from snleee September 9, 2021 21:07
Copy link
Contributor

@snleee snleee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will change our release pipeline steps when building an Apache release. It would be great if you can also update the documentation at https://cwiki.apache.org/confluence/display/PINOT/Creating+an+Apache+Release

@xiangfu0
Copy link
Contributor Author

xiangfu0 commented Sep 9, 2021

This will change our release pipeline steps when building an Apache release. It would be great if you can also update the documentation at https://cwiki.apache.org/confluence/display/PINOT/Creating+an+Apache+Release

will do

@xiangfu0
Copy link
Contributor Author

xiangfu0 commented Sep 9, 2021

fyi: @highker

@xiangfu0 xiangfu0 merged commit 91679c0 into apache:master Sep 9, 2021
@xiangfu0 xiangfu0 deleted the adding-presto-pinot-driver-module branch September 9, 2021 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants