[SPARK-24058][ML][PySpark] Default Params in ML should be saved separately: Python API by viirya · Pull Request #21153 · apache/spark

viirya · 2018-04-25T10:10:40Z

What changes were proposed in this pull request?

See SPARK-23455 for reference. Now default params in ML are saved separately in metadata file in Scala. We must change it for Python for Spark 2.4.0 as well in order to keep them in sync.

How was this patch tested?

Added test.

SparkQA · 2018-04-25T10:14:22Z

Test build #89840 has finished for PR 21153 at commit 0e9b18a.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-04-25T11:01:06Z

Test build #89841 has finished for PR 21153 at commit f32350f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2018-04-25T11:39:03Z

cc @jkbradley @dbtsai

WeichenXu123 · 2018-04-27T08:17:07Z

 jsonParams = paramMap
 else:
 for p in params:
 jsonParams[p.name] = params[p]


I think use _paramMap.copy() will be simpler.

_paramMap's keys are Param not string.

WeichenXu123 · 2018-04-27T08:17:27Z

+ # Default param values
+ jsonDefaultParams = {}
+ for p in instance._defaultParamMap:
+ jsonDefaultParams[p.name] = instance._defaultParamMap[p]


similar, use _defaultParamMap.copy()

WeichenXu123 · 2018-04-27T08:44:48Z

+ for paramName in metadata['defaultParamMap']:
+ param = instance.getParam(paramName)
+ paramValue = metadata['defaultParamMap'][paramName]
+ instance._setDefault(**{param.name: paramValue})


remove line param = instance.getParam(paramName) and change this line to
instance._setDefault(**{paramName: paramValue})

Oh, right. My bad.

SparkQA · 2018-04-27T14:32:55Z

Test build #89924 has finished for PR 21153 at commit 526fa4a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jkbradley

I added a quick comment about moving version utils to a separate PR. I'll try to review the other parts soon. Thanks!

jkbradley · 2018-04-30T22:47:52Z

 return argspec


+def majorMinorVersion(version):


Since this affects Spark core, it might be nice to put this in a separate PR. Also, shall we make this match the Scala API?

Also, shall we make this match the Scala API?
Do you mean to throw exception when unable to parse Spark version?

Yes please, a few things:
throw exception when unable to parse
make it a static method of a class pyspark.util.VersionUtils
add unit test

SparkQA · 2018-05-09T03:09:28Z

Test build #90393 has finished for PR 21153 at commit 86fa433.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-05-09T03:48:54Z

Test build #90394 has finished for PR 21153 at commit b47beab.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jkbradley · 2018-05-11T00:22:20Z

Would you mind rebasing this off of the upstream master branch? I'm having trouble running the tests for this PR locally.

viirya · 2018-05-11T01:07:25Z

Done. Can you try again?

SparkQA · 2018-05-11T01:30:27Z

Test build #90486 has finished for PR 21153 at commit ce84137.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2018-05-14T13:52:55Z

Not sure why AppVeyor build failed. How to re-trigger AppVeyor build?

SparkQA · 2018-05-14T23:07:29Z

Test build #4179 has finished for PR 21153 at commit ce84137.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jkbradley

Sorry for the delay. This LGTM except for the small typo (Hopefully fixing it will re-run the Appveyor test)

jkbradley · 2018-05-09T18:13:48Z

 - sparkVersion
 - uid
 - paramMap
+ - defalutParamMap (since 2.4.0)


typo: default

SparkQA · 2018-05-15T00:31:46Z

Test build #90612 has finished for PR 21153 at commit dc59375.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2018-05-15T02:32:11Z

OK. Looks like AppVeyor build is ok now.

jkbradley · 2018-05-15T23:49:27Z

OK thanks @viirya !
Merging with master

viirya · 2018-05-15T23:50:51Z

Thanks @jkbradley @WeichenXu123

viirya force-pushed the SPARK-24058 branch from 0e9b18a to f32350fCompare April 25, 2018 10:17

Default Params in ML should be saved separately in Python API.
f32350f

WeichenXu123 reviewed Apr 27, 2018
View reviewed changes

Remove redundant call to getParam.
526fa4a

jkbradley reviewed Apr 30, 2018
View reviewed changes

viirya mentioned this pull request May 1, 2018
[SPARK-24131][PySpark] Add majorMinorVersion API to PySpark for determining Spark versions #21203
Closed

Merge remote-tracking branch 'upstream/master' into SPARK-24058
86fa433

Sync with updated majorAndMinorVersions API.
b47beab

Merge remote-tracking branch 'upstream/master' into SPARK-24058
ce84137

jkbradley reviewed May 14, 2018
View reviewed changes

Fix typo.
dc59375

asfgit closed this in 8a13c50May 15, 2018

viirya deleted the SPARK-24058 branch December 27, 2023 18:35

Uh oh!

Conversation

viirya commented Apr 25, 2018

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Apr 25, 2018

Uh oh!

SparkQA commented Apr 25, 2018

Uh oh!

viirya commented Apr 25, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 27, 2018

Uh oh!

jkbradley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viiryaMay 1, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 9, 2018

Uh oh!

SparkQA commented May 9, 2018

Uh oh!

jkbradley commented May 11, 2018

Uh oh!

viirya commented May 11, 2018

Uh oh!

SparkQA commented May 11, 2018

Uh oh!

viirya commented May 14, 2018

Uh oh!

SparkQA commented May 14, 2018

Uh oh!

jkbradley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 15, 2018

Uh oh!

viirya commented May 15, 2018

Uh oh!

jkbradley commented May 15, 2018

Uh oh!

viirya commented May 15, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

viiryaMay 1, 2018•
edited
Loading