Skip to content

[SPARK-9640][Streaming][Test] Do not run Python Kinesis tests when the Kinesis assembly JAR has not been generated #7961

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
tdas wants to merge 11 commits into from
Closed

[SPARK-9640][Streaming][Test] Do not run Python Kinesis tests when the Kinesis assembly JAR has not been generated #7961

tdas wants to merge 11 commits into from

Conversation

@tdas
Copy link
Contributor

@tdas tdas commented Aug 5, 2015

No description provided.

@tdas
Copy link
Contributor Author

tdas commented Aug 5, 2015

@zsxwing Please take a look.

zsxwing
zsxwing reviewed Aug 5, 2015
View reviewed changes
python/pyspark/streaming/tests.py Outdated
Copy link
Member

@zsxwing zsxwing Aug 5, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If ENABLE_KINESIS_TESTS is set to 1 but kinesis_jar_present is false, I think we should fail the test because it should be a bug that the assembly jar cannot be generated correctly.

Copy link
Contributor Author

@tdas tdas Aug 5, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a good point. If that flag is set, then the use definitely wants to run ALL the kinesis tests. In that case its best to fail.

@zsxwing
Copy link
Member

zsxwing commented Aug 5, 2015

I'm worried that if some bug makes the kinesis assembly jar cannot be generated, how to expose it?

@SparkQA
Copy link

SparkQA commented Aug 5, 2015

Test build #39867 has finished for PR 7961 at commit e292e64.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 7, 2015

Test build #1394 has finished for PR 7961 at commit b858196.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

zsxwing
zsxwing reviewed Aug 7, 2015
View reviewed changes
extras/kinesis-asl/pom.xml Outdated
Copy link
Member

@zsxwing zsxwing Aug 7, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You also need to update SparkBuild.scala if you want to change it.

Copy link
Contributor Author

@tdas tdas Aug 7, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right.. forgot that.

@SparkQA
Copy link

SparkQA commented Aug 7, 2015

Test build #40122 has finished for PR 7961 at commit 94be631.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@tdas tdas changed the title [SPARK-9640][Streaming][Test] Do not run Python Kinesis tests when the Kinesis assembly JAR has not been generated [SPARK-9640][SPARK-9727][Streaming][Build][Test] Do not run Python Kinesis tests when the Kinesis assembly JAR has not been generated Aug 7, 2015
zsxwing
zsxwing reviewed Aug 7, 2015
View reviewed changes
python/pyspark/streaming/tests.py Outdated
Copy link
Member

@zsxwing zsxwing Aug 7, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: extra empty line

@SparkQA
Copy link

SparkQA commented Aug 7, 2015

Test build #40128 timed out for PR 7961 at commit ba9cfc7 after a configured wait of 175m.

@tdas
Copy link
Contributor Author

tdas commented Aug 7, 2015

retest this please.

1 similar comment
@tdas
Copy link
Contributor Author

tdas commented Aug 7, 2015

retest this please.

@SparkQA
Copy link

SparkQA commented Aug 7, 2015

Test build #40186 has finished for PR 7961 at commit ba9cfc7.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@tdas
Copy link
Contributor Author

tdas commented Aug 10, 2015

retest this please.

@SparkQA
Copy link

SparkQA commented Aug 10, 2015

Test build #40313 has finished for PR 7961 at commit ba9cfc7.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 10, 2015

Test build #1427 has finished for PR 7961 at commit ca90719.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 11, 2015

Test build #40331 timed out for PR 7961 at commit ca90719 after a configured wait of 175m.

@zsxwing
Copy link
Member

zsxwing commented Aug 11, 2015

LGTM.

@zsxwing
Copy link
Member

zsxwing commented Aug 11, 2015

retest this please

@SparkQA
Copy link

SparkQA commented Aug 11, 2015

Test build #1431 has finished for PR 7961 at commit 9bbb97d.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 11, 2015

Test build #40353 has finished for PR 7961 at commit 004ae26.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class MQTTUtils(object):

@tdas
Copy link
Contributor Author

tdas commented Aug 11, 2015

Because of SPARK-9727, the complete tests are being run. Let me undo changes related to that, and make that a separate PR.

@tdas tdas changed the title [SPARK-9640][SPARK-9727][Streaming][Build][Test] Do not run Python Kinesis tests when the Kinesis assembly JAR has not been generated [SPARK-9640][Streaming][Test] Do not run Python Kinesis tests when the Kinesis assembly JAR has not been generated Aug 11, 2015
@SparkQA
Copy link

SparkQA commented Aug 11, 2015

Test build #1432 has finished for PR 7961 at commit 9bbb97d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 11, 2015

Test build #1436 has finished for PR 7961 at commit 974ce19.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class MQTTUtils(object):

@SparkQA
Copy link

SparkQA commented Aug 11, 2015

Test build #40379 has finished for PR 7961 at commit 974ce19.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zsxwing
Copy link
Member

zsxwing commented Aug 11, 2015

LGTM.

@andrewor14 looks python/unit-tests.log doesn't appear in https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40379/ could you help add it?

@SparkQA
Copy link

SparkQA commented Aug 11, 2015

Test build #40368 timed out for PR 7961 at commit 004ae26 after a configured wait of 175m.

@tdas
Copy link
Contributor Author

tdas commented Aug 11, 2015

retest this please.

@SparkQA
Copy link

SparkQA commented Aug 11, 2015

Test build #1437 has finished for PR 7961 at commit 974ce19.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 11, 2015

Test build #40404 has finished for PR 7961 at commit 974ce19.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class MQTTUtils(object):

@tdas
Copy link
Contributor Author

tdas commented Aug 11, 2015

Alright, I am merging this! Thanks for reviewing @zsxwing . Merging this to master and 1.5

asfgit pushed a commit that referenced this pull request Aug 11, 2015
@tdas
[SPARK-9640] [STREAMING] [TEST] Do not run Python Kinesis tests when …
c7f0090 
…the Kinesis assembly JAR has not been generated

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes #7961 from tdas/SPARK-9640 and squashes the following commits:

974ce19 [Tathagata Das] Undo changes related to SPARK-9727
004ae26 [Tathagata Das] style fixes
9bbb97d [Tathagata Das] Minor style fies
e6a677e [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into SPARK-9640
ca90719 [Tathagata Das] Removed extra line
ba9cfc7 [Tathagata Das] Improved kinesis test selection logic
88d59bd [Tathagata Das] updated test modules
871fcc8 [Tathagata Das] Fixed SparkBuild
94be631 [Tathagata Das] Fixed style
b858196 [Tathagata Das] Fixed conditions and few other things based on PR comments.
e292e64 [Tathagata Das] Added filters for Kinesis python tests

(cherry picked from commit 0f90d60)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
@asfgit asfgit closed this in 0f90d60 Aug 11, 2015
CodingCat pushed a commit to CodingCat/spark that referenced this pull request Aug 17, 2015
@tdas @CodingCat
[SPARK-9640] [STREAMING] [TEST] Do not run Python Kinesis tests when …
a1c44f6 
…the Kinesis assembly JAR has not been generated

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes apache#7961 from tdas/SPARK-9640 and squashes the following commits:

974ce19 [Tathagata Das] Undo changes related to SPARK-9727
004ae26 [Tathagata Das] style fixes
9bbb97d [Tathagata Das] Minor style fies
e6a677e [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into SPARK-9640
ca90719 [Tathagata Das] Removed extra line
ba9cfc7 [Tathagata Das] Improved kinesis test selection logic
88d59bd [Tathagata Das] updated test modules
871fcc8 [Tathagata Das] Fixed SparkBuild
94be631 [Tathagata Das] Fixed style
b858196 [Tathagata Das] Fixed conditions and few other things based on PR comments.
e292e64 [Tathagata Das] Added filters for Kinesis python tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@tdas @zsxwing @SparkQA