Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] SparkBatchProcessBuilder should not set proxy user when keytab is used #3872

Closed
3 of 5 tasks
wangyepeng2 opened this issue Nov 29, 2022 · 2 comments
Closed
3 of 5 tasks

[Bug] SparkBatchProcessBuilder should not set proxy user when keytab is used #3872

wangyepeng2 opened this issue Nov 29, 2022 · 2 comments
Labels
kind:bug This is a clearly a bug priority:major

Comments

@wangyepeng2
Copy link
Contributor

wangyepeng2 commented Nov 29, 2022 •
edited by pan3793
Loading

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the bug

企业微信截图_0b50c8f3-bccd-4c4a-b233-a62646341cb0

The above code is in class SparkBatchProcessBuilder .spark batch process will always set proxy user , even if i set the spark.kerberos.principal. it will cause error:

Only one of --proxy-user or --principal can be provided.
        at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:634)
        at org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:279)
        at org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:234)
        at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:119)
        at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$3.<init>(SparkSubmit.scala:1029)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:1029)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:85)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Affects Version(s)

master

Kyuubi Server Log Output

No response

Kyuubi Engine Log Output

Only one of --proxy-user or --principal can be provided.
        at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:634)
        at org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:279)
        at org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:234)
        at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:119)
        at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$3.<init>(SparkSubmit.scala:1029)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:1029)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:85)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Kyuubi Server Configurations

No response

Kyuubi Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

  • Yes. I can submit a PR independently to fix.
  • Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
  • No. I cannot submit a PR at this time.
@wangyepeng2 wangyepeng2 added kind:bug This is a clearly a bug priority:major labels Nov 29, 2022
@github-actions
Copy link

Hello @wangyepeng2,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi (Incubating).

@pan3793 pan3793 changed the title [Bug] [Bug] SparkBatchProcessBuilder should not set proxy user when keytab is used Nov 29, 2022
@pan3793
Copy link
Member

pan3793 commented Nov 29, 2022

Thanks @wangyepeng2 for reporting this issue, would you mind sending a PR to fix it ?

@wangyepeng2 wangyepeng2 mentioned this issue Mar 30, 2023
Closed
3 tasks
pan3793 added a commit that referenced this issue Mar 30, 2023
@wangyepeng2 @pan3793
[KYUUBI #3872][BATCH] Skip setting proxy user when keytab is provided
d885f19 
### _Why are the changes needed?_

Fix #3872

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4640 from wangyepeng2/proxy_user_fix.

Closes #3872

c7228b9 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala
b6f41d1 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkBatchProcessBuilder.scala
ecf3b3b [wangyepeng2] change function name to setupKerberos
63fc5de [wangyepeng2] fix:spark batch submit should not set proxy user when keytab is used

Lead-authored-by: wangyepeng2 <yepeng.wang@convertlab.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: wangyepeng2 <44561824+wangyepeng2@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit 4c2f1e6)
Signed-off-by: Cheng Pan <chengpan@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind:bug This is a clearly a bug priority:major
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[KYUUBI #3872][BATCH] Skip setting proxy user when keytab is provided wangyepeng2/kyuubi
2 participants
@pan3793 @wangyepeng2