[KYUUBI #5867] HiveEngine support run on YARN mode #5868

yikf · 2023-12-16T12:28:45Z

🔍 Description

Issue References 🔗

This PR aims to support hive engine run on yarn mode, close #5867

Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Types of changes 🔖

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Behavior Without This Pull Request ⚰️

Behavior With This Pull Request 🎉

Related Unit Tests

Checklists

📝 Author Self Checklist

My code follows the style guidelines of this project
I have performed a self-review
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
This patch was not authored or co-authored using Generative Tooling

📝 Committer Pre-Merge Checklist

Be nice. Be informative.

yikf · 2023-12-16T12:49:19Z

@pan3793 Please take a look if you find a time

codecov-commenter · 2023-12-16T13:51:41Z

Codecov Report

Attention: 287 lines in your changes are missing coverage. Please review.

Comparison is base (c6177ab) 61.57% compared to head (44f7287) 61.17%.
Report is 5 commits behind head on master.

Files	Patch %	Lines
...i/engine/deploy/yarn/EngineYarnModeSubmitter.scala	14.97%	171 Missing and 5 partials ⚠️
.../kyuubi/engine/deploy/yarn/ApplicationMaster.scala	7.04%	66 Missing ⚠️
...ngine/deploy/yarn/ApplicationMasterArguments.scala	0.00%	19 Missing ⚠️
...yuubi/engine/hive/HiveYarnModeProcessBuilder.scala	78.94%	15 Missing and 1 partial ⚠️
...apache/kyuubi/engine/hive/HiveProcessBuilder.scala	22.22%	7 Missing ⚠️
...in/scala/org/apache/kyuubi/config/KyuubiConf.scala	98.07%	0 Missing and 1 partial ⚠️
...ain/scala/org/apache/kyuubi/engine/EngineRef.scala	0.00%	1 Missing ⚠️
...pache/kyuubi/engine/KyuubiApplicationManager.scala	80.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #5868      +/-   ##
============================================
- Coverage     61.57%   61.17%   -0.41%     
  Complexity       23       23              
============================================
  Files           616      621       +5     
  Lines         36388    36858     +470     
  Branches       4979     5014      +35     
============================================
+ Hits          22407    22548     +141     
- Misses        11568    11873     +305     
- Partials       2413     2437      +24

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala

kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/EngineYarnModeSubmitter.scala

...-hive-sql-engine/src/main/scala/org/apache/kyuubi/engine/hive/deploy/ApplicationMaster.scala

...e-sql-engine/src/main/scala/org/apache/kyuubi/engine/hive/deploy/HiveYarnModeSubmitter.scala

yikf · 2023-12-21T10:33:06Z

Thanks @cxzl25 , will fix these issues mentioned in your comments.

kyuubi-server/src/main/scala/org/apache/kyuubi/engine/hive/HiveYarnModeProcessBuilder.scala

kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/EngineYarnModeSubmitter.scala

kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala

…arn/EngineYarnModeSubmitter.scala

pan3793 · 2023-12-27T14:38:34Z

Tested locally with https://github.com/awesome-kyuubi/hadoop-testing

overall LGTM, some minor things nice to have (it depends on you to fix it in-place or postpone to the follow-up PRs)

we should support proxy user as spark engine does (should be an independent PR)
can we use the same rule to generate the engine app name
maybe engine type should be KYUUBI ?
set the tags, especially engine_ref_id, so that Kyuubi server could find it

pan3793

I think it's already in a good shape, ready to go

kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala

pan3793 · 2023-12-27T14:41:36Z

kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala

+      .doc(s"kyuubi engine container memory in mb when `${ENGINE_DEPLOY_MODE.key}` is YARN.")
+      .version("1.9.0")
+      .intConf
+      .createWithDefault(1024)


use human-readable format? e.g. 1g, 1024m

Sure, plan support bytesConf in the follow up PR

kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala

kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/ApplicationMaster.scala

…arn/ApplicationMaster.scala

kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/ApplicationMaster.scala

pan3793 · 2023-12-27T15:13:42Z

I just found out we can leverage the YARN mode to address another existing issue of Hive engine: the Apache Hive released binary tarball ships an old version of Scala and Spark jars, we can exclude those jars to avoid potential class conflict issues.

…arn/ApplicationMaster.scala

kyuubi-common/src/main/scala/org/apache/kyuubi/Utils.scala

kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala

kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala

yaooqinn · 2023-12-28T07:24:16Z

Comparison is base (c6177ab) 61.57% compared to head (5a36d6e) 60.96%.

Can we also improve the test coverage?

yikf · 2023-12-28T07:29:17Z

Comparison is base (c6177ab) 61.57% compared to head (5a36d6e) 60.96%.

Can we also improve the test coverage?

Sure

yikf · 2023-12-28T12:21:46Z

Now, it contains the appropriate Name, Application Type and Application Tags

kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala

…arn/EngineYarnModeSubmitter.scala

zhouyifan279 · 2023-12-29T08:38:19Z

LGTM.

zhouyifan279 · 2023-12-29T08:04:50Z

...nals/kyuubi-hive-sql-engine/src/main/scala/org/apache/kyuubi/engine/hive/HiveSQLEngine.scala

@@ -45,7 +45,10 @@ class HiveSQLEngine extends Serverable("HiveSQLEngine") {
    super.start()
    // Start engine self-terminating checker after all services are ready and it can be reached by
    // all servers in engine spaces.
-    backendService.sessionManager.startTerminatingChecker(() => stop())
+    backendService.sessionManager.startTerminatingChecker(() => {
+      selfExist = true


Maybe selfExited ?

pan3793 · 2023-12-29T10:50:51Z

@yikf Thanks, merged to master. we can address the minor comments in follow-up PRs.

# 🔍 Description ## Issue References 🔗 This pull request is a follow-up of #5867, to fix generated configuration docs and typo to recover CI. ## Describe Your Solution 🔧 Regenerated docs by performing `dev/gen/gen_all_config_docs.sh` Address comment #5868 (comment) ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #5932 from pan3793/fix-docs. Closes #5867 93b8f97 [Cheng Pan] docs 0ad0d29 [Cheng Pan] typo Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org>

# 🔍 Description ## Issue References 🔗 This PR aims to support hive engine run on yarn mode, close apache#5867 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [ ] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes apache#5868 from Yikf/hive-on-yarn. Closes apache#5867 44f7287 [yikaifei] fix 3c17d2c [yikaifei] fix test 5474ebf [yikaifei] parse classpath 6b97c42 [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala 34a67b4 [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala 5e5045e [yikaifei] fix app type d1eb5ae [yikaifei] fix d89d09c [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala 1fa18ba [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/ApplicationMaster.scala 1b0b77f [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/ApplicationMaster.scala 2ed1d44 [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala 98ff19c [yikaifei] HiveEngine support run on YARN mode Lead-authored-by: yikaifei <yikaifei@apache.org> Co-authored-by: Cheng Pan <pan3793@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org>

# 🔍 Description ## Issue References 🔗 This pull request is a follow-up of apache#5867, to fix generated configuration docs and typo to recover CI. ## Describe Your Solution 🔧 Regenerated docs by performing `dev/gen/gen_all_config_docs.sh` Address comment apache#5868 (comment) ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes apache#5932 from pan3793/fix-docs. Closes apache#5867 93b8f97 [Cheng Pan] docs 0ad0d29 [Cheng Pan] typo Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org>

github-actions bot added kind:documentation Documentation is a feature! module:server module:hive module:common kind:build module:integration-tests labels Dec 16, 2023

cxzl25 reviewed Dec 18, 2023

View reviewed changes

kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala Outdated Show resolved Hide resolved

cxzl25 reviewed Dec 21, 2023

View reviewed changes

kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/EngineYarnModeSubmitter.scala Outdated Show resolved Hide resolved

cxzl25 reviewed Dec 21, 2023

View reviewed changes

kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/EngineYarnModeSubmitter.scala Outdated Show resolved Hide resolved

cxzl25 reviewed Dec 21, 2023

View reviewed changes

...-hive-sql-engine/src/main/scala/org/apache/kyuubi/engine/hive/deploy/ApplicationMaster.scala Outdated Show resolved Hide resolved

...-hive-sql-engine/src/main/scala/org/apache/kyuubi/engine/hive/deploy/ApplicationMaster.scala Outdated Show resolved Hide resolved

cxzl25 reviewed Dec 21, 2023

View reviewed changes

...e-sql-engine/src/main/scala/org/apache/kyuubi/engine/hive/deploy/HiveYarnModeSubmitter.scala Outdated Show resolved Hide resolved

pan3793 reviewed Dec 26, 2023

View reviewed changes

kyuubi-server/src/main/scala/org/apache/kyuubi/engine/hive/HiveYarnModeProcessBuilder.scala Show resolved Hide resolved

pan3793 reviewed Dec 26, 2023

View reviewed changes

kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/EngineYarnModeSubmitter.scala Outdated Show resolved Hide resolved

pan3793 reviewed Dec 26, 2023

View reviewed changes

kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/EngineYarnModeSubmitter.scala Outdated Show resolved Hide resolved

HiveEngine support run on YARN mode

98ff19c

pan3793 reviewed Dec 27, 2023

View reviewed changes

kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala Outdated Show resolved Hide resolved

Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/y…

2ed1d44

…arn/EngineYarnModeSubmitter.scala

pan3793 approved these changes Dec 27, 2023

View reviewed changes

pan3793 reviewed Dec 27, 2023

View reviewed changes

kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala Outdated Show resolved Hide resolved

pan3793 reviewed Dec 27, 2023

View reviewed changes

kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala Outdated Show resolved Hide resolved

pan3793 reviewed Dec 27, 2023

View reviewed changes

kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/ApplicationMaster.scala Outdated Show resolved Hide resolved

Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/y…

1b0b77f

…arn/ApplicationMaster.scala

pan3793 reviewed Dec 27, 2023

View reviewed changes

kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/ApplicationMaster.scala Outdated Show resolved Hide resolved

Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/y…

1fa18ba

…arn/ApplicationMaster.scala

pan3793 reviewed Dec 28, 2023

View reviewed changes

kyuubi-common/src/main/scala/org/apache/kyuubi/Utils.scala Outdated Show resolved Hide resolved

pan3793 reviewed Dec 28, 2023

View reviewed changes

kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala Outdated Show resolved Hide resolved

yaooqinn reviewed Dec 28, 2023

View reviewed changes

kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala Outdated Show resolved Hide resolved

fix

d1eb5ae

pan3793 reviewed Dec 28, 2023

View reviewed changes

kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala Outdated Show resolved Hide resolved

fix app type

5e5045e

pan3793 reviewed Dec 28, 2023

View reviewed changes

kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala Outdated Show resolved Hide resolved

pan3793 reviewed Dec 28, 2023

View reviewed changes

kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala Outdated Show resolved Hide resolved

pan3793 and others added 5 commits December 28, 2023 20:57

Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/y…

34a67b4

…arn/EngineYarnModeSubmitter.scala

Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/y…

6b97c42

…arn/EngineYarnModeSubmitter.scala

parse classpath

5474ebf

fix test

3c17d2c

fix

44f7287

pan3793 approved these changes Dec 29, 2023

View reviewed changes

pan3793 assigned yikf Dec 29, 2023

pan3793 added this to the v1.9.0 milestone Dec 29, 2023

zhouyifan279 reviewed Dec 29, 2023

View reviewed changes

pan3793 closed this in 679aca5 Dec 29, 2023

pan3793 mentioned this pull request Dec 29, 2023

[KYUUBI #5867][FOLLOWUP] Fix generated configuration docs and typo #5932

Closed

4 tasks

yikf deleted the hive-on-yarn branch January 30, 2024 11:08

pan3793 mentioned this pull request Apr 3, 2024

[TASK][MEDIUM] Support running JDBC engine on YARN AM #6253

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[KYUUBI #5867] HiveEngine support run on YARN mode #5868

[KYUUBI #5867] HiveEngine support run on YARN mode #5868

yikf commented Dec 16, 2023 •

edited by pan3793

Loading

yikf commented Dec 16, 2023

codecov-commenter commented Dec 16, 2023 •

edited

Loading

yikf commented Dec 21, 2023 •

edited

Loading

pan3793 commented Dec 27, 2023 •

edited

Loading

pan3793 left a comment

pan3793 Dec 27, 2023

yikf Dec 28, 2023

pan3793 commented Dec 27, 2023

yaooqinn commented Dec 28, 2023

yikf commented Dec 28, 2023

yikf commented Dec 28, 2023 •

edited

Loading

zhouyifan279 commented Dec 29, 2023 •

edited

Loading

zhouyifan279 Dec 29, 2023 •

edited

Loading

pan3793 commented Dec 29, 2023

[KYUUBI #5867] HiveEngine support run on YARN mode #5868

[KYUUBI #5867] HiveEngine support run on YARN mode #5868

Conversation

yikf commented Dec 16, 2023 • edited by pan3793 Loading

🔍 Description

Issue References 🔗

Describe Your Solution 🔧

Types of changes 🔖

Test Plan 🧪

Behavior Without This Pull Request ⚰️

Behavior With This Pull Request 🎉

Related Unit Tests

Checklists

📝 Author Self Checklist

📝 Committer Pre-Merge Checklist

yikf commented Dec 16, 2023

codecov-commenter commented Dec 16, 2023 • edited Loading

Codecov Report

yikf commented Dec 21, 2023 • edited Loading

pan3793 commented Dec 27, 2023 • edited Loading

pan3793 left a comment

Choose a reason for hiding this comment

pan3793 Dec 27, 2023

Choose a reason for hiding this comment

yikf Dec 28, 2023

Choose a reason for hiding this comment

pan3793 commented Dec 27, 2023

yaooqinn commented Dec 28, 2023

yikf commented Dec 28, 2023

yikf commented Dec 28, 2023 • edited Loading

zhouyifan279 commented Dec 29, 2023 • edited Loading

zhouyifan279 Dec 29, 2023 • edited Loading

Choose a reason for hiding this comment

pan3793 commented Dec 29, 2023

yikf commented Dec 16, 2023 •

edited by pan3793

Loading

codecov-commenter commented Dec 16, 2023 •

edited

Loading

yikf commented Dec 21, 2023 •

edited

Loading

pan3793 commented Dec 27, 2023 •

edited

Loading

yikf commented Dec 28, 2023 •

edited

Loading

zhouyifan279 commented Dec 29, 2023 •

edited

Loading

zhouyifan279 Dec 29, 2023 •

edited

Loading