Skip to content

[SPARK-9572][STREAMING][PYSPARK] Added StreamingContext.getActiveOrCreate() in Python #8080

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
tdas wants to merge 14 commits into from
Closed

[SPARK-9572][STREAMING][PYSPARK] Added StreamingContext.getActiveOrCreate() in Python #8080

tdas wants to merge 14 commits into from

Conversation

@tdas
Copy link
Contributor

@tdas tdas commented Aug 10, 2015

No description provided.

@tdas
Copy link
Contributor Author

tdas commented Aug 10, 2015

@zsxwing Could you take a quick look?

@SparkQA
Copy link

SparkQA commented Aug 10, 2015

Test build #40324 has finished for PR 8080 at commit b711214.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 10, 2015

Test build #1423 has finished for PR 8080 at commit 5781728.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 11, 2015

Test build #1434 has finished for PR 8080 at commit 5781728.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 11, 2015

Test build #40377 has finished for PR 8080 at commit 60479da.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

zsxwing
zsxwing reviewed Aug 11, 2015
View reviewed changes
python/pyspark/streaming/context.py Outdated
Copy link
Member

@zsxwing zsxwing Aug 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to throw an exception when activeJvmContextOption.get().hashCode() != activePythonContextJavaId because it will be confusing that getActive returns None but getActiveOrCreate still complains there is already an active context in Scala codes.

Copy link
Contributor Author

@tdas tdas Aug 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I was on the fence about what to do here. Thanks for deciding for me.

@SparkQA
Copy link

SparkQA commented Aug 11, 2015

Test build #1435 has finished for PR 8080 at commit 9c2da9c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

zsxwing
zsxwing reviewed Aug 11, 2015
View reviewed changes
python/run-tests.py
Copy link
Member

@zsxwing zsxwing Aug 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the previous code means the stdout is in python/unit-tests.log, which is correct.

Copy link
Contributor Author

@tdas tdas Aug 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you see how LOG_FILE is defined, it is already an absolute path.
https://github.com/tdas/spark/blob/SPARK-9572/python/run-tests.py#L69

Copy link
Member

@zsxwing zsxwing Aug 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, right. Good catch about the path.

@zsxwing
Copy link
Member

zsxwing commented Aug 11, 2015

retest this please

@SparkQA
Copy link

SparkQA commented Aug 11, 2015

Test build #40410 has finished for PR 8080 at commit e21488d.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zsxwing
Copy link
Member

zsxwing commented Aug 11, 2015

retest this please

@zsxwing
Copy link
Member

zsxwing commented Aug 11, 2015

LGTM.

@zsxwing
Copy link
Member

zsxwing commented Aug 11, 2015

retest this please

@tdas
Copy link
Contributor Author

tdas commented Aug 11, 2015

retest this please.

@SparkQA
Copy link

SparkQA commented Aug 11, 2015

Test build #40418 has finished for PR 8080 at commit e21488d.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 11, 2015

Test build #40430 has finished for PR 8080 at commit f4f094c.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 11, 2015

Test build #1439 has finished for PR 8080 at commit e21488d.

  • This patch fails Spark unit tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 11, 2015

Test build #40431 has finished for PR 8080 at commit 741a0d0.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

zsxwing
zsxwing reviewed Aug 11, 2015
View reviewed changes
python/pyspark/streaming/tests.py Outdated
Copy link
Member

@zsxwing zsxwing Aug 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

false -> False

Copy link
Contributor Author

@tdas tdas Aug 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldnt this be false based on Java's style of writing false

@SparkQA
Copy link

SparkQA commented Aug 11, 2015

Test build #1446 has started for PR 8080 at commit 741a0d0.

@SparkQA
Copy link

SparkQA commented Aug 11, 2015

Test build #1447 has started for PR 8080 at commit 741a0d0.

@SparkQA
Copy link

SparkQA commented Aug 11, 2015

Test build #40439 has finished for PR 8080 at commit 64a231d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 11, 2015

Test build #1448 has finished for PR 8080 at commit 64a231d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zsxwing
Copy link
Member

zsxwing commented Aug 11, 2015

LGTM

asfgit pushed a commit that referenced this pull request Aug 11, 2015
@tdas
[SPARK-9572] [STREAMING] [PYSPARK] Added StreamingContext.getActiveOr…
71460b8 
…Create() in Python

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes #8080 from tdas/SPARK-9572 and squashes the following commits:

64a231d [Tathagata Das] Fix based on comments
741a0d0 [Tathagata Das] Fixed style
f4f094c [Tathagata Das] Tweaked test
9afcdbe [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into SPARK-9572
e21488d [Tathagata Das] Minor update
1a371d9 [Tathagata Das] Addressed comments.
60479da [Tathagata Das] Fixed indent
9c2da9c [Tathagata Das] Fixed bugs
b5bd32c [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into SPARK-9572
b55b348 [Tathagata Das] Removed prints
5781728 [Tathagata Das] Fix style issues
b711214 [Tathagata Das] Reverted run-tests.py
643b59d [Tathagata Das] Revert unnecessary change
150e58c [Tathagata Das] Added StreamingContext.getActiveOrCreate() in Python

(cherry picked from commit 5b8bb1b)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
@asfgit asfgit closed this in 5b8bb1b Aug 11, 2015
CodingCat pushed a commit to CodingCat/spark that referenced this pull request Aug 17, 2015
@tdas @CodingCat
[SPARK-9572] [STREAMING] [PYSPARK] Added StreamingContext.getActiveOr…
0ad06f1 
…Create() in Python

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes apache#8080 from tdas/SPARK-9572 and squashes the following commits:

64a231d [Tathagata Das] Fix based on comments
741a0d0 [Tathagata Das] Fixed style
f4f094c [Tathagata Das] Tweaked test
9afcdbe [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into SPARK-9572
e21488d [Tathagata Das] Minor update
1a371d9 [Tathagata Das] Addressed comments.
60479da [Tathagata Das] Fixed indent
9c2da9c [Tathagata Das] Fixed bugs
b5bd32c [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into SPARK-9572
b55b348 [Tathagata Das] Removed prints
5781728 [Tathagata Das] Fix style issues
b711214 [Tathagata Das] Reverted run-tests.py
643b59d [Tathagata Das] Revert unnecessary change
150e58c [Tathagata Das] Added StreamingContext.getActiveOrCreate() in Python
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@tdas @SparkQA @zsxwing