Skip to content

[SPARK-8032] [PySpark] Make version checking for NumPy in MLlib more robust #6579

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 5 commits into from
Closed

[SPARK-8032] [PySpark] Make version checking for NumPy in MLlib more robust #6579

wants to merge 5 commits into from

Conversation

MechCoder
Copy link
Contributor

The current checking does version 1.x' is less than1.4' this will fail if x has greater than 1 digit, since x > 4, however 1.x < 1.4

It fails in my system since I have version 1.10 :P

@MechCoder MechCoder changed the title [SPARK-8032] Make version checking for NumPy in MLlib more robust [SPARK-8032] [PySpark] Make version checking for NumPy in MLlib more robust Jun 2, 2015
@MechCoder
Copy link
Contributor Author

ping @mengxr @jkbradley

@SparkQA
Copy link

SparkQA commented Jun 2, 2015

Test build #33968 has finished for PR 6579 at commit 0929877.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@MechCoder MechCoder force-pushed the np_ver branch 2 times, most recently from aae1a2e to 3a3a639 Compare June 2, 2015 14:26
@SparkQA
Copy link

SparkQA commented Jun 2, 2015

Test build #33986 has finished for PR 6579 at commit 3a3a639.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jun 2, 2015

Test build #33989 has finished for PR 6579 at commit 22703dd.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@MechCoder
Copy link
Contributor Author

jenkins test this please

mengxr
mengxr reviewed Jun 2, 2015
View reviewed changes
python/pyspark/mllib/__init__.py
@@ -23,7 +23,14 @@
# MLlib currently needs NumPy 1.4+, so complain if lower

import numpy
if numpy.version.version < '1.4':
ver = numpy.version.version
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this before the official 1.10 release comes out! The following may be simpler:

ver = [int(x) for x in numpy.version.version.split(.)[:2]]
if ver < [1, 4]:
  raise ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this way of list comparison common? Funny, I haven't used it before.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the last section in https://docs.python.org/2/tutorial/datastructures.html

@SparkQA
Copy link

SparkQA commented Jun 2, 2015

Test build #33992 has finished for PR 6579 at commit 22703dd.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@MechCoder
Copy link
Contributor Author

done

@SparkQA
Copy link

SparkQA commented Jun 2, 2015

Test build #34003 has finished for PR 6579 at commit e89376c.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@mengxr
Copy link
Contributor

mengxr commented Jun 2, 2015

@MechCoder Why it is <= instead of <? We do support numpy 1.4.x like 1.4.1.

@MechCoder
Copy link
Contributor Author

I see, I misunderstood the comment '"MLlib requires NumPy 1.4+"

@mengxr
Copy link
Contributor

mengxr commented Jun 2, 2015

@MechCoder please test it on your local and then watch the Jenkins error message. Do not ignore Jenkins because there are flaky tests. Some Jenkins failures are real. For example,

In https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34003/consoleFull, you can see

Running test: pyspark.mllib.classification ... Traceback (most recent call last):
  File "/usr/lib64/python2.6/runpy.py", line 104, in _run_module_as_main
    loader, code, fname = _get_module_details(mod_name)
  File "/usr/lib64/python2.6/runpy.py", line 79, in _get_module_details
    loader = get_loader(mod_name)
  File "/usr/lib64/python2.6/pkgutil.py", line 456, in get_loader
    return find_loader(fullname)
  File "/usr/lib64/python2.6/pkgutil.py", line 466, in find_loader
    for importer in iter_importers(fullname):
  File "/usr/lib64/python2.6/pkgutil.py", line 422, in iter_importers
    __import__(pkg)
  File "pyspark/mllib/__init__.py", line 27
    ver = [int(x) for x in numpy.version.version.split(.)[:2]]

@MechCoder
Copy link
Contributor Author

lol. sorry about the silly mistake.

@SparkQA
Copy link

SparkQA commented Jun 2, 2015

Test build #34004 has finished for PR 6579 at commit e35f0d4.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jun 2, 2015

Test build #34011 has finished for PR 6579 at commit 893fb7e.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jun 2, 2015

Test build #34014 has finished for PR 6579 at commit 15430f8.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

asfgit pushed a commit that referenced this pull request Jun 3, 2015
@MechCoder @mengxr
[SPARK-8032] [PYSPARK] Make version checking for NumPy in MLlib more …
bd57af3 
…robust

The current checking does version `1.x' is less than `1.4' this will fail if x has greater than 1 digit, since x > 4, however `1.x` < `1.4`

It fails in my system since I have version `1.10` :P

Author: MechCoder <manojkumarsivaraj334@gmail.com>

Closes #6579 from MechCoder/np_ver and squashes the following commits:

15430f8 [MechCoder] fix syntax error
893fb7e [MechCoder] remove equal to
e35f0d4 [MechCoder] minor
e89376c [MechCoder] Better checking
22703dd [MechCoder] [SPARK-8032] Make version checking for NumPy in MLlib more robust

(cherry picked from commit 452eb82)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
asfgit pushed a commit that referenced this pull request Jun 3, 2015
@MechCoder @mengxr
[SPARK-8032] [PYSPARK] Make version checking for NumPy in MLlib more …
bbd3772 
…robust

The current checking does version `1.x' is less than `1.4' this will fail if x has greater than 1 digit, since x > 4, however `1.x` < `1.4`

It fails in my system since I have version `1.10` :P

Author: MechCoder <manojkumarsivaraj334@gmail.com>

Closes #6579 from MechCoder/np_ver and squashes the following commits:

15430f8 [MechCoder] fix syntax error
893fb7e [MechCoder] remove equal to
e35f0d4 [MechCoder] minor
e89376c [MechCoder] Better checking
22703dd [MechCoder] [SPARK-8032] Make version checking for NumPy in MLlib more robust

(cherry picked from commit 452eb82)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
asfgit pushed a commit that referenced this pull request Jun 3, 2015
@MechCoder @mengxr
[SPARK-8032] [PYSPARK] Make version checking for NumPy in MLlib more …
23bf307 
…robust

The current checking does version `1.x' is less than `1.4' this will fail if x has greater than 1 digit, since x > 4, however `1.x` < `1.4`

It fails in my system since I have version `1.10` :P

Author: MechCoder <manojkumarsivaraj334@gmail.com>

Closes #6579 from MechCoder/np_ver and squashes the following commits:

15430f8 [MechCoder] fix syntax error
893fb7e [MechCoder] remove equal to
e35f0d4 [MechCoder] minor
e89376c [MechCoder] Better checking
22703dd [MechCoder] [SPARK-8032] Make version checking for NumPy in MLlib more robust

(cherry picked from commit 452eb82)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
@asfgit asfgit closed this in 452eb82 Jun 3, 2015
asfgit pushed a commit that referenced this pull request Jun 3, 2015
@MechCoder @mengxr
[SPARK-8032] [PYSPARK] Make version checking for NumPy in MLlib more …
36eed2f 
…robust

The current checking does version `1.x' is less than `1.4' this will fail if x has greater than 1 digit, since x > 4, however `1.x` < `1.4`

It fails in my system since I have version `1.10` :P

Author: MechCoder <manojkumarsivaraj334@gmail.com>

Closes #6579 from MechCoder/np_ver and squashes the following commits:

15430f8 [MechCoder] fix syntax error
893fb7e [MechCoder] remove equal to
e35f0d4 [MechCoder] minor
e89376c [MechCoder] Better checking
22703dd [MechCoder] [SPARK-8032] Make version checking for NumPy in MLlib more robust

(cherry picked from commit 452eb82)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
asfgit pushed a commit that referenced this pull request Jun 3, 2015
@MechCoder @mengxr
[SPARK-8032] [PYSPARK] Make version checking for NumPy in MLlib more …
fed98b9 
…robust

The current checking does version `1.x' is less than `1.4' this will fail if x has greater than 1 digit, since x > 4, however `1.x` < `1.4`

It fails in my system since I have version `1.10` :P

Author: MechCoder <manojkumarsivaraj334@gmail.com>

Closes #6579 from MechCoder/np_ver and squashes the following commits:

15430f8 [MechCoder] fix syntax error
893fb7e [MechCoder] remove equal to
e35f0d4 [MechCoder] minor
e89376c [MechCoder] Better checking
22703dd [MechCoder] [SPARK-8032] Make version checking for NumPy in MLlib more robust

(cherry picked from commit 452eb82)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
@mengxr
Copy link
Contributor

mengxr commented Jun 3, 2015

LGTM. Merged into master and all branches since 1.0. Thanks!

@MechCoder MechCoder deleted the np_ver branch June 3, 2015 12:05
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@MechCoder @jeanlyn
[SPARK-8032] [PYSPARK] Make version checking for NumPy in MLlib more …
01125b4 
…robust

The current checking does version `1.x' is less than `1.4' this will fail if x has greater than 1 digit, since x > 4, however `1.x` < `1.4`

It fails in my system since I have version `1.10` :P

Author: MechCoder <manojkumarsivaraj334@gmail.com>

Closes apache#6579 from MechCoder/np_ver and squashes the following commits:

15430f8 [MechCoder] fix syntax error
893fb7e [MechCoder] remove equal to
e35f0d4 [MechCoder] minor
e89376c [MechCoder] Better checking
22703dd [MechCoder] [SPARK-8032] Make version checking for NumPy in MLlib more robust
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@MechCoder @nemccarthy
[SPARK-8032] [PYSPARK] Make version checking for NumPy in MLlib more …
105f8aa 
…robust

The current checking does version `1.x' is less than `1.4' this will fail if x has greater than 1 digit, since x > 4, however `1.x` < `1.4`

It fails in my system since I have version `1.10` :P

Author: MechCoder <manojkumarsivaraj334@gmail.com>

Closes apache#6579 from MechCoder/np_ver and squashes the following commits:

15430f8 [MechCoder] fix syntax error
893fb7e [MechCoder] remove equal to
e35f0d4 [MechCoder] minor
e89376c [MechCoder] Better checking
22703dd [MechCoder] [SPARK-8032] Make version checking for NumPy in MLlib more robust
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@MechCoder @SparkQA @mengxr