Skip to content

[SPARK-6327] [PySpark] fix launch spark-submit from python #5019

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

davies
Copy link
Contributor

@davies davies commented Mar 13, 2015

SparkSubmit should be launched without setting PYSPARK_SUBMIT_ARGS

cc @JoshRosen , this mode is actually used by python unit test, so I will not add more test for it.

@SparkQA
Copy link

SparkQA commented Mar 13, 2015

Test build #28591 has started for PR 5019 at commit 2c20b0c.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Mar 13, 2015

Test build #28591 has finished for PR 5019 at commit 2c20b0c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28591/
Test PASSed.

@vanzin
Copy link
Contributor

vanzin commented Mar 14, 2015

LGTM.

@jerryshao
Copy link
Contributor

Hi @vanzin @davies , seems this change hides the pyspark-shell into the python code, so basically if we want to add some additional arguments with PYSPARK_SUBMIT_ARGS, do we need to override this environment variable as what I mentioned in #3916 , or how to handle this? Thanks a lot.

@vanzin
Copy link
Contributor

vanzin commented Mar 16, 2015

@jerryshao you should be able to just set that variable. Doesn't that work for you? Davies's fix actually allows you to do that (before it would be overridden by the pyspark script).

@davies
Copy link
Contributor Author

davies commented Mar 16, 2015

@jerryshao You can still PYSPARK_SUBMIT_ARGS by having pyspark-shell in the end. It's not perfect, but it's fine for a internal things.

@jerryshao
Copy link
Contributor

OK, get it, thanks a lot.

@JoshRosen
Copy link
Contributor

I was able to reproduce this bug locally and have verified that it's fixed by this patch. LGTM, so I'm going to merge this into master (1.4.0). Thanks!

@asfgit asfgit closed this in e3f315a Mar 16, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants