[SPARK-15402] [ML] [PySpark] PySpark ml.evaluation should support save/load #13194
Conversation
|
Test build #58860 has finished for PR 13194 at commit
|
|
Test build #58864 has finished for PR 13194 at commit
|
| from pyspark.context import SparkContext | ||
| from pyspark.sql import SQLContext | ||
| globs = globals().copy() | ||
| globs = pyspark.ml.evaluation.__dict__.copy() |
There was a problem hiding this comment.
can you explain why you need this change (and add a comment about it)?
There was a problem hiding this comment.
@thunterdb If we did not make this change, we can work well when using ./bin/pyspark to run all tests, but it failed when run with ./python/run-tests --modules=pyspark-ml.This issue is weird, I still not figure out the root cause, and we did the same changes when we add other save/load.
|
Test build #59204 has finished for PR 13194 at commit
|
|
This seems pretty reasonable, do you have the time to update it to master and then maybe we could get @MLnick to take a quick look? |
yanboliang
force-pushed
the
spark-15402
branch
from
cd0c9d7 to
8764e83
Compare
|
Test build #66665 has finished for PR 13194 at commit
|
|
Resolved merge conflicts, ping @holdenk @MLnick @srowen @jkbradley to take a look when you available. This is just add Python API and should be very straight forward to move ahead. Thanks. |
| @@ -21,7 +21,8 @@ | |||
| from pyspark.ml.wrapper import JavaParams | |||
| from pyspark.ml.param import Param, Params, TypeConverters | |||
| from pyspark.ml.param.shared import HasLabelCol, HasPredictionCol, HasRawPredictionCol | |||
| from pyspark.ml.common import inherit_doc | |||
| from pyspark.ml.util import JavaMLReadable, JavaMLWritable | |||
| from pyspark.mllib.common import inherit_doc | |||
There was a problem hiding this comment.
Just wondering why change to import from mllib.common?
|
Just one question, otherwise LGTM |
| globs=globs, optionflags=doctest.ELLIPSIS) | ||
| spark.stop() | ||
| if failure_count: | ||
| exit(-1) |
There was a problem hiding this comment.
should there still be an exit(-1) when failures? If not then you can remove the returned variables from doctest.testmod
There was a problem hiding this comment.
Thanks for remind, added check failure_count.
|
Test build #66827 has finished for PR 13194 at commit
|
|
Since this is very straight forward, I'll merge it into master. Any comments are still appreciated, and I can address them in follow up work. Thanks for review! @holdenk @BryanCutler |
…load ## What changes were proposed in this pull request? Since ```ml.evaluation``` has supported save/load at Scala side, supporting it at Python side is very straightforward and easy. ## How was this patch tested? Add python doctest. Author: Yanbo Liang <ybliang8@gmail.com> Closes apache#13194 from yanboliang/spark-15402.
…load ## What changes were proposed in this pull request? Since ```ml.evaluation``` has supported save/load at Scala side, supporting it at Python side is very straightforward and easy. ## How was this patch tested? Add python doctest. Author: Yanbo Liang <ybliang8@gmail.com> Closes apache#13194 from yanboliang/spark-15402.
What changes were proposed in this pull request?
Since
ml.evaluationhas supported save/load at Scala side, supporting it at Python side is very straightforward and easy.How was this patch tested?
Add python doctest.