[SPARK-2871] [PySpark] Add missing API #1791

davies · 2014-08-05T17:51:13Z

Add the following APIs:

SparkContext.conf
SparkContext.isLocal
SparkContext.startTime

SparkQA · 2014-08-05T22:33:29Z

QA tests have started for PR 1791. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17953/consoleFull

SparkQA · 2014-08-05T23:33:53Z

QA results for PR 1791:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17953/consoleFull

SparkQA · 2014-08-05T23:54:23Z

QA tests have started for PR 1791. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17958/consoleFull

SparkQA · 2014-08-05T23:58:22Z

QA tests have started for PR 1791. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17959/consoleFull

SparkQA · 2014-08-06T00:48:23Z

QA results for PR 1791:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17958/consoleFull

zzl0 · 2014-08-06T05:20:44Z

core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala

+  }
+
+  /**
+   * Convert a RDD of Java objects to and RDD of serialized Python objects, that is usable by


Convert a RDD of Java objects to and RDD of serialized Python objects
=>
Convert an RDD of Java objects to an RDD of serialized Python objects ?

SparkQA · 2014-08-06T05:44:27Z

QA tests have started for PR 1791. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18001/consoleFull

davies · 2014-08-06T06:19:17Z

The histogram() had been implemented in pure Python, it will support integer better, also it will support RDD of strings and other comparable objects.

This was inspired by #1783 et, and much improved.

SparkQA · 2014-08-06T06:48:51Z

QA results for PR 1791:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18001/consoleFull

nrchandan · 2014-08-06T06:50:12Z

python/pyspark/rdd.py

+
+            inc = (maxv - minv) / buckets
+            # keep them as integer if possible
+            if inc * buckets != maxv - minv:


This was smart!

SparkQA · 2014-08-06T18:04:21Z

QA tests have started for PR 1791. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18032/consoleFull

SparkQA · 2014-08-06T18:53:01Z

QA results for PR 1791:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18032/consoleFull

meanApprox() and sumApprox()

SparkQA · 2014-08-06T21:39:29Z

QA tests have started for PR 1791. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18055/consoleFull

SparkQA · 2014-08-06T21:40:27Z

QA results for PR 1791:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class BoundedFloat(float):

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18055/consoleFull

SparkQA · 2014-08-06T21:48:33Z

QA tests have started for PR 1791. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18058/consoleFull

davies · 2014-08-14T00:17:54Z

Jenkins, test this please

SparkQA · 2014-08-14T00:20:33Z

QA tests have started for PR 1791. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18498/consoleFull

SparkQA · 2014-08-14T01:11:40Z

QA results for PR 1791:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class BoundedFloat(float):

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18498/consoleFull

mateiz · 2014-08-14T21:08:58Z

python/pyspark/rdd.py

@@ -812,23 +842,39 @@ def func(iterator):

        return self.mapPartitions(func).fold(zeroValue, combOp)

-    def max(self):
+    def max(self, comp=None):


Maybe explain what "comp" is in the doc comment

mateiz · 2014-08-14T21:25:31Z

@davies I looked over all of this now and made some comments, but you should have Josh check too. Just to be clear though, I don't think this can make it into 1.1, so we can hold off on it for a while while we fix issues for 1.1. But these are great APIs to have.

1. implement lookup(), similar to that in Scala 2. handle None, nan, inf in histogram, add many tests 3. remove collectPartitions() 4. improve docs

davies · 2014-08-14T23:03:48Z

@mateiz thanks for review this, I had addressed all you comments.

@JoshRosen could you take a look a this again?

davies · 2014-08-14T23:25:28Z

The description had been updated to list all the added APIs.

Conflicts: python/pyspark/rdd.py python/pyspark/tests.py

davies · 2014-08-22T05:25:17Z

@mateiz @JoshRosen some APIs has been splitted out as separated PRs: #2091, #2092, #2093, #2094, #2095

Conflicts: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala python/pyspark/rdd.py

SparkQA · 2014-08-26T18:20:59Z

QA tests have started for PR 1791 at commit 28fd368.

This patch merges cleanly.

SparkQA · 2014-08-26T18:31:02Z

QA tests have started for PR 1791 at commit 1ac98d6.

This patch merges cleanly.

SparkQA · 2014-08-26T19:11:06Z

QA tests have started for PR 1791 at commit 657a09b.

This patch merges cleanly.

SparkQA · 2014-08-26T19:12:09Z

QA tests have finished for PR 1791 at commit 28fd368.

This patch fails unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class BoundedFloat(float):

SparkQA · 2014-08-26T19:22:45Z

QA tests have finished for PR 1791 at commit 1ac98d6.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-08-26T19:57:10Z

QA tests have finished for PR 1791 at commit 657a09b.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

davies · 2014-08-27T21:58:13Z

Most of useful parts have been merged separately, so close this.

davies added 4 commits August 4, 2014 23:55

add missing API in SparkContext

ff2cbe3

add histogram()

e0b3d30

change histogram API

5d5be95

add zipWithIndex and zipWithUniqueId

a95eca0

collectPartitions()

4ffae00

update docs of histogram

7a9ea0a

davies changed the title ~~[WIP] [PySpark] Add missing API~~ [PySpark] Add missing API Aug 5, 2014

davies changed the title ~~[PySpark] Add missing API~~ [SPARK-2871] [PySpark] Add missing API Aug 6, 2014

zzl0 reviewed Aug 6, 2014
View reviewed changes

davies added 2 commits August 5, 2014 22:38

histogram() in pure Python, better support for int

53640be

fix docs

9a01ac3

nrchandan mentioned this pull request Aug 6, 2014

[SPARK-1170] Add histogram method to Python's RDD API #1783

Closed

nrchandan reviewed Aug 6, 2014
View reviewed changes

refactor

7ba5f88

davies added 2 commits August 6, 2014 14:33

fix bug of countApproxDistinct

a25c34e

add countApprox and countApproxDistinct

1218b3b

meanApprox() and sumApprox()

Merge branch 'master' into api

034124f

mateiz reviewed Aug 14, 2014
View reviewed changes

address all the comments:

63c013d

1. implement lookup(), similar to that in Scala 2. handle None, nan, inf in histogram, add many tests 3. remove collectPartitions() 4. improve docs

Merge branch 'master' into api

1213aca

Conflicts: python/pyspark/rdd.py python/pyspark/tests.py

Merge branch 'master' into api

28fd368

Conflicts: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala python/pyspark/rdd.py

remove splitted changes

1ac98d6

remove countApproxDistinct()

657a09b

davies closed this Aug 27, 2014

davies mentioned this pull request Aug 27, 2014

[SPARK-2871] [PySpark] add countApproxDistinct() API #2142

Closed

davies deleted the api branch September 15, 2014 22:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-2871] [PySpark] Add missing API #1791

[SPARK-2871] [PySpark] Add missing API #1791

davies commented Aug 5, 2014

SparkQA commented Aug 5, 2014

SparkQA commented Aug 5, 2014

SparkQA commented Aug 5, 2014

SparkQA commented Aug 5, 2014

SparkQA commented Aug 6, 2014

zzl0 Aug 6, 2014

SparkQA commented Aug 6, 2014

davies commented Aug 6, 2014

SparkQA commented Aug 6, 2014

nrchandan Aug 6, 2014

SparkQA commented Aug 6, 2014

SparkQA commented Aug 6, 2014

SparkQA commented Aug 6, 2014

SparkQA commented Aug 6, 2014

SparkQA commented Aug 6, 2014

davies commented Aug 14, 2014

SparkQA commented Aug 14, 2014

SparkQA commented Aug 14, 2014

mateiz Aug 14, 2014

mateiz commented Aug 14, 2014

davies commented Aug 14, 2014

davies commented Aug 14, 2014

davies commented Aug 22, 2014

SparkQA commented Aug 26, 2014

SparkQA commented Aug 26, 2014

SparkQA commented Aug 26, 2014

SparkQA commented Aug 26, 2014

SparkQA commented Aug 26, 2014

SparkQA commented Aug 26, 2014

davies commented Aug 27, 2014

[SPARK-2871] [PySpark] Add missing API #1791

[SPARK-2871] [PySpark] Add missing API #1791

Conversation

davies commented Aug 5, 2014

SparkQA commented Aug 5, 2014

SparkQA commented Aug 5, 2014

SparkQA commented Aug 5, 2014

SparkQA commented Aug 5, 2014

SparkQA commented Aug 6, 2014

zzl0 Aug 6, 2014

Choose a reason for hiding this comment

SparkQA commented Aug 6, 2014

davies commented Aug 6, 2014

SparkQA commented Aug 6, 2014

nrchandan Aug 6, 2014

Choose a reason for hiding this comment

SparkQA commented Aug 6, 2014

SparkQA commented Aug 6, 2014

SparkQA commented Aug 6, 2014

SparkQA commented Aug 6, 2014

SparkQA commented Aug 6, 2014

davies commented Aug 14, 2014

SparkQA commented Aug 14, 2014

SparkQA commented Aug 14, 2014

mateiz Aug 14, 2014

Choose a reason for hiding this comment

mateiz commented Aug 14, 2014

davies commented Aug 14, 2014

davies commented Aug 14, 2014

davies commented Aug 22, 2014

SparkQA commented Aug 26, 2014

SparkQA commented Aug 26, 2014

SparkQA commented Aug 26, 2014

SparkQA commented Aug 26, 2014

SparkQA commented Aug 26, 2014

SparkQA commented Aug 26, 2014

davies commented Aug 27, 2014