[SPARK-11938][ML] Expose numFeatures in all ML PredictionModel for Py… by Lewuathe · Pull Request #9936 · apache/spark

Lewuathe · 2015-11-24T15:30:49Z

…Spark

see: https://issues.apache.org/jira/browse/SPARK-11938

Note: Test cases for ml.regression and ml.classification were not implemented. The test written in this patch target only for numFeatures. We can do enhancement of test case for ml.regression and ml.classification in separate JIRA.

SparkQA · 2015-11-24T16:05:38Z

Test build #46609 has finished for PR 9936 at commit 25475ba.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Lewuathe · 2015-11-28T01:10:42Z

@mengxr Could you review it? Thanks.

sethah · 2015-11-30T18:20:11Z

"The number of features" -> "The number of features used to train the model."
I think this is more explicit since, for things like decision trees, this property is what number of features were used in training and not necessarily what number of features the model incorporates in its prediction.
Nit: add period to follow docstring convention.

@sethah Thank you. I updated!

SparkQA · 2015-12-02T02:39:59Z

Test build #47029 has finished for PR 9936 at commit 478e113.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yanboliang · 2015-12-09T11:44:28Z

Could we test numFeatures at each model that we can save cost of creating DataFrame and training model?

Each trainer output corresponding models. So we cannot share training code but creating DataFrames can be separated each RegressorTest and ClassificationTest. I'll update so.

@Lewuathe Sorry for late response. What I means is that we can add test for numFeatures in the current test cases of each algorithm. For example, we can add
>>> model.numFeatures 2
after here for LogisticRegressionModel.
The python doctest can also used as examples that users can get how to use numFeatures from the test case context.

Lewuathe · 2015-12-12T02:42:59Z

@yanboliang I made test code to avoid recreation of datasets.
Could you check again please?

SparkQA · 2015-12-12T02:49:27Z

Test build #47604 has finished for PR 9936 at commit 3b02d4e.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-12-12T03:20:29Z

Test build #47605 has finished for PR 9936 at commit 5b4c500.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

thunterdb · 2016-01-08T00:00:36Z

is there any reason for putting the import in the code?

Vectors and StringIndexer is not used in any other place. It is better not to expand the scope in my though.

I think either is fine. I'm sure Vectors will be used elsewhere in this file in the future, but am not sure about StringIndexer.

SparkQA · 2016-01-23T04:21:22Z

Test build #49926 has finished for PR 9936 at commit cb8904e.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-01-23T06:47:48Z

Test build #49928 has finished for PR 9936 at commit 2ac0329.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

holdenk · 2016-01-28T19:00:24Z

It seems like there is a lot of duplicated just calls to Java & PyDoc could we maybe make a HasNumFeatures or similar (similar to how we do this for params which we use in many places)?

jkbradley · 2016-03-29T21:55:35Z

Ping: Is this PR still active? (It would be nice)

yanboliang · 2016-03-31T15:45:33Z

@Lewuathe Do you have time to update this PR? If not, I can help.

Lewuathe · 2016-04-01T10:51:57Z

Sorry for late response. I'll do update. @yanboliang Thanks for taking care!

Aggregate numFeatures property in HasNumFeaturesModel in base.py.

SparkQA · 2016-04-01T14:33:15Z

Test build #54704 has finished for PR 9936 at commit b211e3d.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-01T14:38:14Z

Test build #54705 has finished for PR 9936 at commit 3f718d0.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class LogisticRegressionModel(HasNumFeaturesModel, JavaModel, JavaMLWritable, JavaMLReadable):
- class DecisionTreeClassificationModel(HasNumFeaturesModel, DecisionTreeModel, JavaMLWritable, JavaMLReadable):
- class RandomForestClassificationModel(HasNumFeaturesModel, TreeEnsembleModels):
- class GBTClassificationModel(HasNumFeaturesModel, TreeEnsembleModels):
- class NaiveBayesModel(HasNumFeaturesModel, JavaModel, JavaMLWritable, JavaMLReadable):
- class MultilayerPerceptronClassificationModel(HasNumFeaturesModel, JavaModel, JavaMLWritable, JavaMLReadable):
- class LinearRegressionModel(HasNumFeaturesModel, JavaModel, JavaMLWritable, JavaMLReadable):
- class DecisionTreeRegressionModel(HasNumFeaturesModel, DecisionTreeModel, JavaMLWritable, JavaMLReadable):
- class RandomForestRegressionModel(HasNumFeaturesModel, TreeEnsembleModels):
- class GBTRegressionModel(HasNumFeaturesModel, TreeEnsembleModels):

SparkQA · 2016-04-02T02:30:38Z

Test build #54749 has finished for PR 9936 at commit a128635.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DecisionTreeClassificationModel(HasNumFeaturesModel, DecisionTreeModel, JavaMLWritable,
- class MultilayerPerceptronClassificationModel(HasNumFeaturesModel, JavaModel, JavaMLWritable,
- class DecisionTreeRegressionModel(HasNumFeaturesModel, DecisionTreeModel, JavaMLWritable,

jkbradley · 2016-04-20T20:25:35Z

@Lewuathe Things got busy over here too; sorry for the delay! Btw, can you please add "[PYTHON]" to the PR title?

jkbradley · 2016-04-20T20:25:38Z

+ It should be mixin with JavaModel.
+ """
+ @property
+ @since("1.7.0")


Remove since tag. If this is added to another class in a later version, the since tag will be incorrect.

SparkQA · 2016-05-12T11:50:31Z

Test build #58466 has finished for PR 9936 at commit a128635.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
- class DecisionTreeClassificationModel(HasNumFeaturesModel, DecisionTreeModel, JavaMLWritable,
- class MultilayerPerceptronClassificationModel(HasNumFeaturesModel, JavaModel, JavaMLWritable,
- class DecisionTreeRegressionModel(HasNumFeaturesModel, DecisionTreeModel, JavaMLWritable,

Lewuathe force-pushed the SPARK-11938 branch from eaf149a to 25475baCompare November 24, 2015 15:32

sethah reviewed Nov 30, 2015
View reviewed changes

yanboliang reviewed Dec 9, 2015
View reviewed changes

thunterdb reviewed Jan 8, 2016
View reviewed changes

Lewuathe force-pushed the SPARK-11938 branch from cb8904e to 2ac0329Compare January 23, 2016 06:17

Lewuathe force-pushed the SPARK-11938 branch from 2ac0329 to b211e3dCompare April 1, 2016 14:31

[SPARK-11938] Expose numFeatures in all ML PredictionModel for PySpark
3f718d0
Aggregate numFeatures property in HasNumFeaturesModel in base.py.

Lewuathe force-pushed the SPARK-11938 branch from b211e3d to 3f718d0Compare April 1, 2016 14:32

[SPARK-11938] Fix style
a128635

jkbradley reviewed Apr 20, 2016
View reviewed changes

Uh oh!

Conversation

Lewuathe commented Nov 24, 2015

Uh oh!

SparkQA commented Nov 24, 2015

Uh oh!

Lewuathe commented Nov 28, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 2, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lewuathe commented Dec 12, 2015

Uh oh!

SparkQA commented Dec 12, 2015

Uh oh!

SparkQA commented Dec 12, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 23, 2016

Uh oh!

SparkQA commented Jan 23, 2016

Uh oh!

holdenk commented Jan 28, 2016

Uh oh!

jkbradley commented Mar 29, 2016

Uh oh!

yanboliang commented Mar 31, 2016

Uh oh!

Lewuathe commented Apr 1, 2016

Uh oh!

SparkQA commented Apr 1, 2016

Uh oh!

SparkQA commented Apr 1, 2016

Uh oh!

SparkQA commented Apr 2, 2016

Uh oh!

jkbradley commented Apr 20, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 12, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants