[SPARK-11938][ML] Expose numFeatures in all ML PredictionModel for Py… #9936

Lewuathe · 2015-11-24T15:30:49Z

…Spark

see: https://issues.apache.org/jira/browse/SPARK-11938

Note: Test cases for ml.regression and ml.classification were not implemented. The test written in this patch target only for numFeatures. We can do enhancement of test case for ml.regression and ml.classification in separate JIRA.

SparkQA · 2015-11-24T16:05:38Z

Test build #46609 has finished for PR 9936 at commit 25475ba.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Lewuathe · 2015-11-28T01:10:42Z

@mengxr Could you review it? Thanks.

sethah · 2015-11-30T18:20:11Z

python/pyspark/ml/classification.py

+    @since("1.6.0")
+    def numFeatures(self):
+        """
+        The number of features


"The number of features" -> "The number of features used to train the model."

I think this is more explicit since, for things like decision trees, this property is what number of features were used in training and not necessarily what number of features the model incorporates in its prediction.

Nit: add period to follow docstring convention.

@sethah Thank you. I updated!

SparkQA · 2015-12-02T02:39:59Z

Test build #47029 has finished for PR 9936 at commit 478e113.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yanboliang · 2015-12-09T11:44:28Z

python/pyspark/ml/tests.py

@@ -371,6 +378,126 @@ def test_fit_maximize_metric(self):
        self.assertEqual(1.0, bestModelMetric, "Best model has R-squared of 1")


+class RegressorTest(PySparkTestCase):


Could we test numFeatures at each model that we can save cost of creating DataFrame and training model?

Each trainer output corresponding models. So we cannot share training code but creating DataFrames can be separated each RegressorTest and ClassificationTest. I'll update so.

@Lewuathe Sorry for late response. What I means is that we can add test for numFeatures in the current test cases of each algorithm. For example, we can add

>>> model.numFeatures 2

after here for LogisticRegressionModel.
The python doctest can also used as examples that users can get how to use numFeatures from the test case context.

Lewuathe · 2015-12-12T02:42:59Z

@yanboliang I made test code to avoid recreation of datasets.
Could you check again please?

SparkQA · 2015-12-12T02:49:27Z

Test build #47604 has finished for PR 9936 at commit 3b02d4e.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-12-12T03:20:29Z

Test build #47605 has finished for PR 9936 at commit 5b4c500.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

thunterdb · 2016-01-08T00:00:36Z

python/pyspark/ml/tests.py

+        try:
+            self.df
+        except AttributeError:
+            from pyspark.mllib.linalg import Vectors


is there any reason for putting the import in the code?

Vectors and StringIndexer is not used in any other place. It is better not to expand the scope in my though.

I think either is fine. I'm sure Vectors will be used elsewhere in this file in the future, but am not sure about StringIndexer.

SparkQA · 2016-01-23T04:21:22Z

Test build #49926 has finished for PR 9936 at commit cb8904e.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-01-23T06:47:48Z

Test build #49928 has finished for PR 9936 at commit 2ac0329.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

holdenk · 2016-01-28T19:00:24Z

It seems like there is a lot of duplicated just calls to Java & PyDoc could we maybe make a HasNumFeatures or similar (similar to how we do this for params which we use in many places)?

jkbradley · 2016-03-29T21:55:35Z

Ping: Is this PR still active? (It would be nice)

yanboliang · 2016-03-31T15:45:33Z

@Lewuathe Do you have time to update this PR? If not, I can help.

Lewuathe · 2016-04-01T10:51:57Z

Sorry for late response. I'll do update. @yanboliang Thanks for taking care!

Aggregate numFeatures property in HasNumFeaturesModel in base.py.

SparkQA · 2016-04-01T14:33:15Z

Test build #54704 has finished for PR 9936 at commit b211e3d.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-01T14:38:14Z

Test build #54705 has finished for PR 9936 at commit 3f718d0.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class LogisticRegressionModel(HasNumFeaturesModel, JavaModel, JavaMLWritable, JavaMLReadable):
- class DecisionTreeClassificationModel(HasNumFeaturesModel, DecisionTreeModel, JavaMLWritable, JavaMLReadable):
- class RandomForestClassificationModel(HasNumFeaturesModel, TreeEnsembleModels):
- class GBTClassificationModel(HasNumFeaturesModel, TreeEnsembleModels):
- class NaiveBayesModel(HasNumFeaturesModel, JavaModel, JavaMLWritable, JavaMLReadable):
- class MultilayerPerceptronClassificationModel(HasNumFeaturesModel, JavaModel, JavaMLWritable, JavaMLReadable):
- class LinearRegressionModel(HasNumFeaturesModel, JavaModel, JavaMLWritable, JavaMLReadable):
- class DecisionTreeRegressionModel(HasNumFeaturesModel, DecisionTreeModel, JavaMLWritable, JavaMLReadable):
- class RandomForestRegressionModel(HasNumFeaturesModel, TreeEnsembleModels):
- class GBTRegressionModel(HasNumFeaturesModel, TreeEnsembleModels):

SparkQA · 2016-04-02T02:30:38Z

Test build #54749 has finished for PR 9936 at commit a128635.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DecisionTreeClassificationModel(HasNumFeaturesModel, DecisionTreeModel, JavaMLWritable,
- class MultilayerPerceptronClassificationModel(HasNumFeaturesModel, JavaModel, JavaMLWritable,
- class DecisionTreeRegressionModel(HasNumFeaturesModel, DecisionTreeModel, JavaMLWritable,

jkbradley · 2016-04-20T20:25:35Z

@Lewuathe Things got busy over here too; sorry for the delay! Btw, can you please add "[PYTHON]" to the PR title?

jkbradley · 2016-04-20T20:25:38Z

python/pyspark/ml/base.py

+    It should be mixin with JavaModel.
+    """
+    @property
+    @since("1.7.0")


Remove since tag. If this is added to another class in a later version, the since tag will be incorrect.

SparkQA · 2016-05-12T11:50:31Z

Test build #58466 has finished for PR 9936 at commit a128635.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
- class DecisionTreeClassificationModel(HasNumFeaturesModel, DecisionTreeModel, JavaMLWritable,
- class MultilayerPerceptronClassificationModel(HasNumFeaturesModel, JavaModel, JavaMLWritable,
- class DecisionTreeRegressionModel(HasNumFeaturesModel, DecisionTreeModel, JavaMLWritable,

vectorijk · 2016-06-24T10:23:39Z

ping @Lewuathe Is this still active? If not, I could help with this.

Lewuathe · 2016-06-26T02:43:53Z

@vectorijk It seems difficult to handle this JIRA for me. Could you help me with this?
Thanks so much for taking care!

vectorijk · 2016-06-27T07:59:35Z

@Lewuathe Thanks! I opened a new PR #13922 here for this issue. Would you mind closing this PR later? And feel free to give comments to that PR.

Closing the following PRs due to requests or unresponsive users. Closes apache#13923 Closes apache#14462 Closes apache#13123 Closes apache#14423 (requested by srowen) Closes apache#14424 (requested by srowen) Closes apache#14101 (requested by jkbradley) Closes apache#10676 (requested by srowen) Closes apache#10943 (requested by yhuai) Closes apache#9936 Closes apache#10701

Lewuathe force-pushed the SPARK-11938 branch from eaf149a to 25475ba Compare November 24, 2015 15:32

sethah reviewed Nov 30, 2015
View reviewed changes

yanboliang reviewed Dec 9, 2015
View reviewed changes

thunterdb reviewed Jan 8, 2016
View reviewed changes

Lewuathe force-pushed the SPARK-11938 branch from cb8904e to 2ac0329 Compare January 23, 2016 06:17

Lewuathe force-pushed the SPARK-11938 branch from 2ac0329 to b211e3d Compare April 1, 2016 14:31

[SPARK-11938] Expose numFeatures in all ML PredictionModel for PySpark

3f718d0

Aggregate numFeatures property in HasNumFeaturesModel in base.py.

Lewuathe force-pushed the SPARK-11938 branch from b211e3d to 3f718d0 Compare April 1, 2016 14:32

[SPARK-11938] Fix style

a128635

jkbradley reviewed Apr 20, 2016
View reviewed changes

vanzin mentioned this pull request Aug 4, 2016

MAINTENANCE. Cleaning up stale PRs. #14495

Closed

asfgit closed this in 53e766c Aug 4, 2016

		@@ -371,6 +378,126 @@ def test_fit_maximize_metric(self):
		self.assertEqual(1.0, bestModelMetric, "Best model has R-squared of 1")


		class RegressorTest(PySparkTestCase):

[SPARK-11938][ML] Expose numFeatures in all ML PredictionModel for Py… #9936

[SPARK-11938][ML] Expose numFeatures in all ML PredictionModel for Py… #9936

Uh oh!

Conversation

Lewuathe commented Nov 24, 2015

Uh oh!

SparkQA commented Nov 24, 2015

Uh oh!

Lewuathe commented Nov 28, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 2, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lewuathe commented Dec 12, 2015

Uh oh!

SparkQA commented Dec 12, 2015

Uh oh!

SparkQA commented Dec 12, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 23, 2016

Uh oh!

SparkQA commented Jan 23, 2016

Uh oh!

holdenk commented Jan 28, 2016

Uh oh!

jkbradley commented Mar 29, 2016

Uh oh!

yanboliang commented Mar 31, 2016

Uh oh!

Lewuathe commented Apr 1, 2016

Uh oh!

SparkQA commented Apr 1, 2016

Uh oh!

SparkQA commented Apr 1, 2016

Uh oh!

SparkQA commented Apr 2, 2016

Uh oh!

jkbradley commented Apr 20, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 12, 2016

Uh oh!

vectorijk commented Jun 24, 2016

Uh oh!

Lewuathe commented Jun 26, 2016

Uh oh!

vectorijk commented Jun 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

vectorijk commented Jun 27, 2016 •

edited

Loading