[SPARK-11171][SPARK-11237][SPARK-11241][ML] Try adding PMMLExportable to ML with KMeans by holdenk · Pull Request #9207 · apache/spark

holdenk · 2015-10-22T00:02:24Z

Add PMMLExportable to ML & use it for KMeans in ML pipeline. This is a bit different than the PMMLExportable in MLLib since the default trait doesn't go through a factory but instead depends on the model implementing it itself (and at the start that will mostly just be calling the parent model's method).
Still a WIP but wanted to see if anyone had comments about the slight change.

SparkQA · 2015-10-22T00:10:02Z

Test build #44104 has finished for PR 9207 at commit 368879f.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * trait PMMLExportable\n

SparkQA · 2015-10-22T00:56:55Z

Test build #44105 has finished for PR 9207 at commit 1749aec.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * trait PMMLExportable\n

SparkQA · 2015-10-22T01:51:36Z

Test build #44111 has finished for PR 9207 at commit bc1b508.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * trait PMMLExportable\n

… only (since actual PMML model evaluation is to be done through a spark-packages project and previous JIRA decided re-loading PMML is out of project scope)

holdenk · 2015-10-23T18:12:02Z

cc @jkbradley

SparkQA · 2015-10-23T18:39:46Z

Test build #44241 has finished for PR 9207 at commit adf0b36.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * trait PMMLExportable\n

holdenk · 2015-10-29T06:22:50Z

re-ping @jkbradley

holdenk · 2015-11-08T23:53:11Z

re-ping @jkbradley

holdenk · 2015-11-08T23:53:33Z

Also maybe @dbtsai if you have a chance to look at this that would be great.

holdenk · 2015-12-01T20:55:29Z

So I've updated this against master @jkbradley

SparkQA · 2015-12-01T21:49:28Z

Test build #46990 has finished for PR 9207 at commit 494ecbf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dbtsai · 2015-12-11T04:03:37Z

Seems this is a copy-paste of org.apache.spark.mllib.pmml, should we deprecate the mllib one, and use the new one in ml package?

I think that might be good, the main difference is this avoids using the factory implementation that the MLLib API was.

(Uses the same public facing API as per the JIRA discussion re: lack of complaints from users with old API)

holdenk · 2015-12-16T20:17:42Z

ping @jkbradley

…r-ML-KMeans

SparkQA · 2015-12-30T19:24:09Z

Test build #48499 has finished for PR 9207 at commit 41611b8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-01-11T22:06:16Z

Test build #49179 has finished for PR 9207 at commit 9525283.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

holdenk · 2016-01-12T05:16:41Z

jenkins retest this please

SparkQA · 2016-01-12T05:32:01Z

Test build #49214 has finished for PR 9207 at commit 461c1ce.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

…r-ML-KMeans

SparkQA · 2016-01-21T00:42:59Z

Test build #49819 has finished for PR 9207 at commit b514421.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

holdenk · 2016-01-21T01:24:06Z

Jenkins retest this please

SparkQA · 2016-01-21T02:17:43Z

Test build #49837 has finished for PR 9207 at commit b514421.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-03T09:20:27Z

Test build #61683 has finished for PR 9207 at commit 49f8a8d.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-03T10:03:08Z

Test build #61684 has finished for PR 9207 at commit e6845f1.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

…r-ML-KMeans

…ing change

SparkQA · 2016-07-05T21:40:58Z

Test build #61784 has finished for PR 9207 at commit 9170b3f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

holdenk · 2016-07-05T22:47:42Z

HiveSubmit failures seem unrelated, jenkins retest this please.

SparkQA · 2016-07-06T00:49:40Z

Test build #61799 has finished for PR 9207 at commit 9170b3f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-18T20:57:39Z

Test build #62478 has finished for PR 9207 at commit 8579c1b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

holdenk · 2016-07-25T21:44:26Z

Now that 2.0 RC5 has passed - maybe it would be an OK time to revisit this? I think it can also be an ok base for adding more formats if we need to.

…r-ML-KMeans

… 2.1

SparkQA · 2016-08-01T20:27:39Z

Test build #63086 has finished for PR 9207 at commit 8103b76.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…r-ML-KMeans

SparkQA · 2016-08-04T00:45:34Z

Test build #63193 has finished for PR 9207 at commit bdcfbd1.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Least(children: Seq[Expression]) extends Expression
- case class Greatest(children: Seq[Expression]) extends Expression
- implicit class SchemaAttribute(f: StructField)

SparkQA · 2016-08-06T01:40:50Z

Test build #63295 has finished for PR 9207 at commit 00173aa.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class ShuffleIndexInformation
- public class ShuffleIndexRecord
- case class CreateTable(tableDesc: CatalogTable, mode: SaveMode, query: Option[LogicalPlan])
- case class PreprocessDDL(conf: SQLConf) extends Rule[LogicalPlan]

holdenk · 2016-08-08T19:45:59Z

ping @MLnick now that 2.0 is out for awhile.

…r-ML-KMeans

SparkQA · 2016-09-09T00:21:02Z

Test build #65123 has finished for PR 9207 at commit 0e8c523.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

holdenk · 2016-11-16T12:15:35Z

Ping @jkbradley / @MLnick - we are coming up on a year since we told people we would add PMML export to Spark in about a year so I figured I'd try and update this again. I'll update this PR tonight against the latest master.

SparkQA · 2016-11-16T19:07:44Z

Test build #68724 has finished for PR 9207 at commit 9cb8994.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

holdenk · 2017-09-21T20:09:32Z

@MLnick: Do you have the bandwith to revisit this? I'm open to refactoring to a more plug-gable approach if we've got the review bandwidth for it.

SparkQA · 2017-09-23T05:51:15Z

Test build #82102 has finished for PR 9207 at commit 9cb8994.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

holdenk · 2018-03-05T19:57:30Z

Closing this in favor of the approach taken in #19876

Try adding PMMLExportable to ML with KMeans
1749aec

holdenk force-pushed the SPARK-11171-SPARK-11237-Add-PMML-export-for-ML-KMeans branch from 368879f to 1749aecCompare October 22, 2015 00:13

Everything is better with tests
bc1b508

Move the PMML loading utils into test and make it clear they are test…
adf0b36
… only (since actual PMML model evaluation is to be done through a spark-packages project and previous JIRA decided re-loading PMML is out of project scope)

holdenk changed the title ~~[SPARK-11171][SPARK-11237][SPARK-11241][ML][WIP] Try adding PMMLExportable to ML with KMeans~~[SPARK-11171][SPARK-11237][SPARK-11241][ML] Try adding PMMLExportable to ML with KMeansOct 23, 2015

Merge in master (now both pmml and native export)
494ecbf

dbtsai reviewed Dec 11, 2015
View reviewed changes

Merge branch 'master' into SPARK-11171-SPARK-11237-Add-PMML-export-fo…
41611b8
…r-ML-KMeans

Merge in master
9525283

Fix import ordering from automerge
461c1ce

holdenk added 2 commits January 20, 2016 12:38

Merge branch 'master' into SPARK-11171-SPARK-11237-Add-PMML-export-fo…
9aa5265
…r-ML-KMeans

Import order fix
b514421

Add pmml() to the writer
49f8a8d

pmmml -> pmml
e6845f1

holdenk added 2 commits July 5, 2016 12:40

Merge branch 'master' into SPARK-11171-SPARK-11237-Add-PMML-export-fo…
0b042e8
…r-ML-KMeans

Add a MIMA exclusion for the saveImpl change and fix inadvertant spac…
9170b3f
…ing change

Merge in master
8579c1b

holdenk added 2 commits August 1, 2016 11:11

Merge branch 'master' into SPARK-11171-SPARK-11237-Add-PMML-export-fo…
c8573f0
…r-ML-KMeans

Remove TODO since we have moved it to Spark 2.1 now that master is on…
8103b76
… 2.1

Merge branch 'master' into SPARK-11171-SPARK-11237-Add-PMML-export-fo…
bdcfbd1
…r-ML-KMeans

Update master
00173aa

Merge branch 'master' into SPARK-11171-SPARK-11237-Add-PMML-export-fo…
0e8c523
…r-ML-KMeans

Merge in master
9cb8994

holdenk mentioned this pull request Dec 6, 2017
[ML][SPARK-23783][SPARK-11239] Add PMML export to Spark ML pipelines #19876
Closed

holdenk closed this Mar 5, 2018

Uh oh!

Conversation

holdenk commented Oct 22, 2015

Uh oh!

SparkQA commented Oct 22, 2015

Uh oh!

SparkQA commented Oct 22, 2015

Uh oh!

SparkQA commented Oct 22, 2015

Uh oh!

holdenk commented Oct 23, 2015

Uh oh!

SparkQA commented Oct 23, 2015

Uh oh!

holdenk commented Oct 29, 2015

Uh oh!

holdenk commented Nov 8, 2015

Uh oh!

holdenk commented Nov 8, 2015

Uh oh!

holdenk commented Dec 1, 2015

Uh oh!

SparkQA commented Dec 1, 2015

Uh oh!

dbtsaiDec 11, 2015

Choose a reason for hiding this comment

Uh oh!

holdenkDec 11, 2015

Choose a reason for hiding this comment

Uh oh!

holdenkDec 11, 2015

Choose a reason for hiding this comment

Uh oh!

holdenk commented Dec 16, 2015

Uh oh!

SparkQA commented Dec 30, 2015

Uh oh!

SparkQA commented Jan 11, 2016

Uh oh!

holdenk commented Jan 12, 2016

Uh oh!

SparkQA commented Jan 12, 2016

Uh oh!

SparkQA commented Jan 21, 2016

Uh oh!

holdenk commented Jan 21, 2016

Uh oh!

SparkQA commented Jan 21, 2016

Uh oh!

SparkQA commented Jul 3, 2016

Uh oh!

SparkQA commented Jul 3, 2016

Uh oh!

SparkQA commented Jul 5, 2016

Uh oh!

holdenk commented Jul 5, 2016

Uh oh!

SparkQA commented Jul 6, 2016

Uh oh!

SparkQA commented Jul 18, 2016

Uh oh!

holdenk commented Jul 25, 2016

Uh oh!

SparkQA commented Aug 1, 2016

Uh oh!

SparkQA commented Aug 4, 2016

Uh oh!

SparkQA commented Aug 6, 2016

Uh oh!

holdenk commented Aug 8, 2016

Uh oh!

SparkQA commented Sep 9, 2016

Uh oh!

holdenk commented Nov 16, 2016

Uh oh!

SparkQA commented Nov 16, 2016

Uh oh!

holdenk commented Sep 21, 2017

Uh oh!

SparkQA commented Sep 23, 2017