Uh oh!
There was an error while loading. Please reload this page.
[SPARK-11237][ML] Add pmml export for k-means in Spark ML - #20907
[SPARK-11237][ML] Add pmml export for k-means in Spark ML#20907holdenk wants to merge 5 commits into
Conversation
holdenk
commented
Mar 26, 2018
SparkQA
commented
Mar 26, 2018
Test build #88599 has finished for PR 20907 at commit
|
| // Save model data: cluster centers | ||
| val data: Array[ClusterData] = instance.clusterCenters.zipWithIndex.map { | ||
| case (center, idx) => | ||
| ClusterData(idx, center) |
There was a problem hiding this comment.
doesn't this type change Data -> ClusterData change the schema of the output parquet file?
There was a problem hiding this comment.
I'm not 100% sure. I'll manually test we can load the old format first.
There was a problem hiding this comment.
Wait no this shouldn't change anything, were saving this with a DataFrame and the schema is the same.
See the schema from 1: res3: org.apache.spark.sql.types.StructType = StructType(StructField(clusterIdx,IntegerType,false), StructField(clusterCenter,org.apache.spark.ml.linalg.VectorUDT@3bfc3ba7,true)) and the new one org.apache.spark.sql.types.StructType = StructType(StructField(clusterIdx,IntegerType,false), StructField(clusterCenter,org.apache.spark.ml.linalg.VectorUDT@3bfc3ba7,true))
| // model. | ||
| val pmmlClusteringModel = pmml.getModels.get(0).asInstanceOf[ClusteringModel] | ||
| assert(pmmlClusteringModel.getNumberOfClusters === clusterCenters.length) | ||
| } |
There was a problem hiding this comment.
Isn't this missing a call to testPMMLWrite?
There was a problem hiding this comment.
Oh yeah :( Thanks for catching that.
SparkQA
commented
Mar 30, 2018
Test build #88766 has finished for PR 20907 at commit
|
SparkQA
commented
Apr 2, 2018
Test build #88823 has finished for PR 20907 at commit
|
holdenk
commented
Apr 4, 2018
Yup, this is ready. If you think its good feel free to merge it otherwise I'll merge it during my regular Friday review time :) |
viirya
commented
Apr 23, 2018
LGTM |
One question I have is, how do users know if a model (e.g. KMeansModel after this change) supports pmml & internal formats without looking into source code? I did a search on the current docs, but didn't find any info. |
dbtsai
commented
Apr 23, 2018
LGTM too! +1 on the documentation which can be a followup PR. Merged into master, and thanks. DB Tsai | Siri Open Source Technologies | Apple, Inc |
What changes were proposed in this pull request?
Adding PMML export to Spark ML's KMeans Model.
How was this patch tested?
New unit test for Spark ML PMML export based on the old Spark MLlib unit test.