Skip to content

[SPARK-7202] [MLlib] [PySpark] Add SparseMatrixPickler to SerDe #5775

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[SPARK-7202] [MLlib] [PySpark] Add SparseMatrixPickler to SerDe #5775

wants to merge 1 commit into from

Conversation

MechCoder
Copy link
Contributor

Utilities for pickling and unpickling SparseMatrices using SerDe

@MechCoder MechCoder changed the title [SPARK-7202] Add SparseMatrixPickler to SerDe [SPARK-7202] [MLlib] [PySpark] Add SparseMatrixPickler to SerDe Apr 29, 2015
@MechCoder
Copy link
Contributor Author

ping @jkbradley @mengxr

@SparkQA
Copy link

SparkQA commented Apr 29, 2015

Test build #31271 has finished for PR 5775 at commit 59fcf52.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
  • This patch adds the following new dependencies:
    • spark-unsafe_2.10-1.4.0-SNAPSHOT.jar

@SparkQA
Copy link

SparkQA commented Apr 29, 2015

Test build #31273 has finished for PR 5775 at commit 25a1819.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
  • This patch adds the following new dependencies:
    • spark-unsafe_2.10-1.4.0-SNAPSHOT.jar

This was referenced May 1, 2015
Closed
Closed
@MechCoder
Copy link
Contributor Author

@mengxr Can you please have a look at this? Thanks in advance! :)

@mengxr
Copy link
Contributor

mengxr commented May 5, 2015

LGTM. Let's call Jenkins another time.

@MechCoder
Copy link
Contributor Author

test this please

@SparkQA
Copy link

SparkQA commented May 5, 2015

Test build #31848 has finished for PR 5775 at commit 25a1819.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 5, 2015

Test build #31853 has finished for PR 5775 at commit 6c59a9a.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@MechCoder
Copy link
Contributor Author

@mengxr I'm not sure what the test failure is about..

@MechCoder
Copy link
Contributor Author

test this please

@SparkQA
Copy link

SparkQA commented May 5, 2015

Test build #31857 has finished for PR 5775 at commit 6c59a9a.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 5, 2015

Test build #31860 has finished for PR 5775 at commit 7e689dc.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@MechCoder
Copy link
Contributor Author

test this please

@SparkQA
Copy link

SparkQA commented May 5, 2015

Test build #31870 has finished for PR 5775 at commit 7e689dc.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

asfgit pushed a commit that referenced this pull request May 5, 2015
@MechCoder @mengxr
[SPARK-7202] [MLLIB] [PYSPARK] Add SparseMatrixPickler to SerDe
cd55e9a 
Utilities for pickling and unpickling SparseMatrices using SerDe

Author: MechCoder <manojkumarsivaraj334@gmail.com>

Closes #5775 from MechCoder/spark-7202 and squashes the following commits:

7e689dc [MechCoder] [SPARK-7202] Add SparseMatrixPickler to SerDe

(cherry picked from commit 5ab652c)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
@mengxr
Copy link
Contributor

mengxr commented May 5, 2015

Merged into master and branch-1.4. Thanks!

@asfgit asfgit closed this in 5ab652c May 5, 2015
@MechCoder MechCoder deleted the spark-7202 branch May 5, 2015 15:55
asfgit pushed a commit that referenced this pull request May 7, 2015
@MechCoder @jkbradley
[SPARK-7328] [MLLIB] [PYSPARK] Pyspark.mllib.linalg.Vectors: Missing …
4436e26 
…items

Add
1. Class methods squared_dist
3. parse
4. norm
5. numNonzeros
6. copy

I made a few vectorizations wrt squared_dist and dot as well. I have added support for SparseMatrix serialization in a separate PR (#5775) and plan to complete support for Matrices in another PR.

Author: MechCoder <manojkumarsivaraj334@gmail.com>

Closes #5872 from MechCoder/local_linalg_api and squashes the following commits:

a8ff1e0 [MechCoder] minor
ce3e53e [MechCoder] Add error message for parser
1bd3c04 [MechCoder] Robust parser and removed unnecessary methods
f779561 [MechCoder] [SPARK-7328] Pyspark.mllib.linalg.Vectors: Missing items

(cherry picked from commit 347a329)
Signed-off-by: Joseph K. Bradley <joseph@databricks.com>
asfgit pushed a commit that referenced this pull request May 7, 2015
@MechCoder @jkbradley
[SPARK-7328] [MLLIB] [PYSPARK] Pyspark.mllib.linalg.Vectors: Missing …
347a329 
…items

Add
1. Class methods squared_dist
3. parse
4. norm
5. numNonzeros
6. copy

I made a few vectorizations wrt squared_dist and dot as well. I have added support for SparseMatrix serialization in a separate PR (#5775) and plan to complete support for Matrices in another PR.

Author: MechCoder <manojkumarsivaraj334@gmail.com>

Closes #5872 from MechCoder/local_linalg_api and squashes the following commits:

a8ff1e0 [MechCoder] minor
ce3e53e [MechCoder] Add error message for parser
1bd3c04 [MechCoder] Robust parser and removed unnecessary methods
f779561 [MechCoder] [SPARK-7328] Pyspark.mllib.linalg.Vectors: Missing items
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@MechCoder @jeanlyn
[SPARK-7202] [MLLIB] [PYSPARK] Add SparseMatrixPickler to SerDe
d054f97 
Utilities for pickling and unpickling SparseMatrices using SerDe

Author: MechCoder <manojkumarsivaraj334@gmail.com>

Closes apache#5775 from MechCoder/spark-7202 and squashes the following commits:

7e689dc [MechCoder] [SPARK-7202] Add SparseMatrixPickler to SerDe
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@MechCoder @jeanlyn
[SPARK-7328] [MLLIB] [PYSPARK] Pyspark.mllib.linalg.Vectors: Missing …
c341596 
…items

Add
1. Class methods squared_dist
3. parse
4. norm
5. numNonzeros
6. copy

I made a few vectorizations wrt squared_dist and dot as well. I have added support for SparseMatrix serialization in a separate PR (apache#5775) and plan to complete support for Matrices in another PR.

Author: MechCoder <manojkumarsivaraj334@gmail.com>

Closes apache#5872 from MechCoder/local_linalg_api and squashes the following commits:

a8ff1e0 [MechCoder] minor
ce3e53e [MechCoder] Add error message for parser
1bd3c04 [MechCoder] Robust parser and removed unnecessary methods
f779561 [MechCoder] [SPARK-7328] Pyspark.mllib.linalg.Vectors: Missing items
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@MechCoder @jeanlyn
[SPARK-7202] [MLLIB] [PYSPARK] Add SparseMatrixPickler to SerDe
201aad3 
Utilities for pickling and unpickling SparseMatrices using SerDe

Author: MechCoder <manojkumarsivaraj334@gmail.com>

Closes apache#5775 from MechCoder/spark-7202 and squashes the following commits:

7e689dc [MechCoder] [SPARK-7202] Add SparseMatrixPickler to SerDe
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@MechCoder @jeanlyn
[SPARK-7328] [MLLIB] [PYSPARK] Pyspark.mllib.linalg.Vectors: Missing …
aec3402 
…items

Add
1. Class methods squared_dist
3. parse
4. norm
5. numNonzeros
6. copy

I made a few vectorizations wrt squared_dist and dot as well. I have added support for SparseMatrix serialization in a separate PR (apache#5775) and plan to complete support for Matrices in another PR.

Author: MechCoder <manojkumarsivaraj334@gmail.com>

Closes apache#5872 from MechCoder/local_linalg_api and squashes the following commits:

a8ff1e0 [MechCoder] minor
ce3e53e [MechCoder] Add error message for parser
1bd3c04 [MechCoder] Robust parser and removed unnecessary methods
f779561 [MechCoder] [SPARK-7328] Pyspark.mllib.linalg.Vectors: Missing items
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@MechCoder @nemccarthy
[SPARK-7202] [MLLIB] [PYSPARK] Add SparseMatrixPickler to SerDe
7856c73 
Utilities for pickling and unpickling SparseMatrices using SerDe

Author: MechCoder <manojkumarsivaraj334@gmail.com>

Closes apache#5775 from MechCoder/spark-7202 and squashes the following commits:

7e689dc [MechCoder] [SPARK-7202] Add SparseMatrixPickler to SerDe
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@MechCoder @nemccarthy
[SPARK-7328] [MLLIB] [PYSPARK] Pyspark.mllib.linalg.Vectors: Missing …
9650a57 
…items

Add
1. Class methods squared_dist
3. parse
4. norm
5. numNonzeros
6. copy

I made a few vectorizations wrt squared_dist and dot as well. I have added support for SparseMatrix serialization in a separate PR (apache#5775) and plan to complete support for Matrices in another PR.

Author: MechCoder <manojkumarsivaraj334@gmail.com>

Closes apache#5872 from MechCoder/local_linalg_api and squashes the following commits:

a8ff1e0 [MechCoder] minor
ce3e53e [MechCoder] Add error message for parser
1bd3c04 [MechCoder] Robust parser and removed unnecessary methods
f779561 [MechCoder] [SPARK-7328] Pyspark.mllib.linalg.Vectors: Missing items
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@MechCoder @SparkQA @mengxr