[SPARK-18693][ML][MLLIB] ML Evaluators should use weight column by imatiach-msft · Pull Request #16557 · apache/spark

imatiach-msft · 2017-01-12T06:00:46Z

What changes were proposed in this pull request?

The evaluators BinaryClassificationEvaluator, RegressionEvaluator, and MulticlassClassificationEvaluator and the corresponding metrics classes BinaryClassificationMetrics, RegressionMetrics and MulticlassMetrics should use sample weight data.
The updates to the regression metrics were based on (and updated with new changes based on comments):
https://issues.apache.org/jira/browse/SPARK-11520
("RegressionMetrics should support instance weights")
but the pull request was closed as the changes were never checked in.

How was this patch tested?

This is still a work in progress, I will be adding more tests soon. I took the regression tests from:
#9907
Which was closed as a stale PR but I updated it with some changes.

SparkQA · 2017-01-12T06:05:15Z

Test build #71247 has finished for PR 16557 at commit 2b624c0.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class BinaryClassificationMetrics @Since(\"2.2.0\") (
class MulticlassMetrics @Since(\"1.1.0\") (predAndLabelsWithOptWeight: RDD[_])

SparkQA · 2017-01-12T17:59:39Z

Test build #71270 has finished for PR 16557 at commit f6d6fef.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

imatiach-msft · 2017-01-12T18:09:21Z

it looks like a random test timed out:
org.apache.spark.scheduler.BasicSchedulerIntegrationSuite.job with fetch failure
Error Details
java.util.concurrent.TimeoutException: Futures timed out after [1 second]
I will try to run again.

imatiach-msft · 2017-01-12T18:09:31Z

Jenkins retest this please

imatiach-msft · 2017-01-12T18:10:24Z

Jenkins, retest this please

SparkQA · 2017-01-12T20:23:58Z

Test build #71272 has finished for PR 16557 at commit f6d6fef.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-01-13T00:49:50Z

Test build #71278 has finished for PR 16557 at commit 54fe922.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-01-13T05:32:44Z

Test build #71291 has started for PR 16557 at commit 397c26b.

imatiach-msft · 2017-01-13T15:14:52Z

Jenkins, retest this please

imatiach-msft · 2017-01-13T15:49:33Z

Jenkins, retest this please

imatiach-msft · 2017-01-13T15:55:54Z

Jenkins, retest this please

imatiach-msft · 2017-01-13T15:56:58Z

Jenkins doesn't seem to be working ...

imatiach-msft · 2017-01-13T16:06:15Z

@sethah @Lewuathe @thunterdb @WeichenXu123 @jkbradley would you be able to take a look at the changes to add a weight column to binary/multiclass/regression evaluators/metrics classes? It looks like you are familiar with this code. Thank you!

imatiach-msft · 2017-01-13T17:14:29Z

Jenkins, retest this please

SparkQA · 2017-01-13T18:28:56Z

Test build #71331 has finished for PR 16557 at commit 808ca6b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class BinaryClassificationMetrics @Since(\"2.2.0\") (
class MulticlassMetrics @Since(\"1.1.0\") (predAndLabelsWithOptWeight: RDD[_])

SparkQA · 2017-01-13T19:42:17Z

Test build #71334 has finished for PR 16557 at commit 808ca6b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class BinaryClassificationMetrics @Since(\"2.2.0\") (
class MulticlassMetrics @Since(\"1.1.0\") (predAndLabelsWithOptWeight: RDD[_])

imatiach-msft · 2017-01-17T18:55:41Z

Jenkins, retest this please

SparkQA · 2017-01-17T21:31:35Z

Test build #71523 has finished for PR 16557 at commit 228bbfb.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class BinaryClassificationMetrics @Since(\"2.2.0\") (
class MulticlassMetrics @Since(\"1.1.0\") (predAndLabelsWithOptWeight: RDD[_])

SparkQA · 2017-01-17T23:59:21Z

Test build #71531 has finished for PR 16557 at commit e2873f2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

imatiach-msft · 2017-01-18T00:14:51Z

Jenkins, retest this please

SparkQA · 2017-01-18T00:19:55Z

Test build #71544 has finished for PR 16557 at commit e2873f2.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

imatiach-msft · 2017-01-18T02:39:14Z

Jenkins, retest this please

imatiach-msft · 2017-01-18T14:49:07Z

Jenkins, retest this please

SparkQA · 2017-01-18T17:22:13Z

Test build #71604 has finished for PR 16557 at commit e2873f2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-01-18T18:13:28Z

Test build #71607 has finished for PR 16557 at commit d589dfe.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-01-19T20:05:16Z

Test build #71670 has finished for PR 16557 at commit c1f5f09.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class SparkListenerExecutorBlacklisted(
case class SparkListenerExecutorUnblacklisted(time: Long, executorId: String)
case class SparkListenerNodeBlacklisted(
case class SparkListenerNodeUnblacklisted(time: Long, hostId: String)
case class QualifiedTableName(database: String, name: String)
class FindHiveSerdeTable(session: SparkSession) extends Rule[LogicalPlan]

imatiach-msft · 2017-01-23T16:29:12Z

Jenkins, retest this please

imatiach-msft · 2017-01-23T16:29:42Z

ping @sethah @Lewuathe @thunterdb @WeichenXu123 @jkbradley would you be able to take a look at the changes to add a weight column to binary/multiclass/regression evaluators/metrics classes? It looks like you are familiar with this code. Thank you!

SparkQA · 2017-01-23T16:34:41Z

Test build #71858 has finished for PR 16557 at commit ba68f72.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-01-23T19:12:35Z

Test build #71860 has finished for PR 16557 at commit 6dbb0ad.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

imatiach-msft · 2017-01-25T21:06:01Z

ping @sethah @Lewuathe @thunterdb @WeichenXu123 @jkbradley would you be able to take a look at the changes to add a weight column to binary/multiclass/regression evaluators/metrics classes? It looks like you are familiar with this code. Thank you!

imatiach-msft · 2017-01-26T14:26:35Z

ping @sethah @Lewuathe @thunterdb @WeichenXu123 @jkbradley would you be able to take a look at the changes to add a weight column to binary/multiclass/regression evaluators/metrics classes? It looks like you are familiar with this code. Thank you!

imatiach-msft · 2017-01-26T15:47:52Z

@srowen @yanboliang might you be able to take a look at this PR? Is it possibly too large and I should break it up into 3 PRs, one per evaluator/metrics class?

srowen · 2017-01-26T16:12:12Z

I wouldn't ping that frequently, please. I don't feel qualified to review this myself.

sethah · 2017-01-26T21:10:05Z

+1 for breaking it up, maybe starting with regression. Also, just because something hasn't been reviewed in two weeks does not mean that there is no interest in it. Two weeks is not all that long (I've seen valuable PRs sit for much longer than that) and it likely just means people are busy. As Sean pointed out, it's probably not necessary to ping once per day.

thunterdb · 2017-02-17T19:08:35Z

I agree, let's break this PR. It will go faster, and some changes may require longer discussions.

imatiach-msft · 2017-02-27T18:11:50Z

ok, I will close this and create three new PRs, one for each of the evaluators

imatiach-msft · 2017-02-27T18:32:18Z

I've created 3 PRs, located here:
#17084
#17085
#17086

SparkQA · 2017-02-27T20:39:04Z

Test build #73525 has finished for PR 16557 at commit a0fc4c3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…ed weight column for multiclass classification evaluator ## What changes were proposed in this pull request? The evaluators BinaryClassificationEvaluator, RegressionEvaluator, and MulticlassClassificationEvaluator and the corresponding metrics classes BinaryClassificationMetrics, RegressionMetrics and MulticlassMetrics should use sample weight data. I've closed the PR: #16557 as recommended in favor of creating three pull requests, one for each of the evaluators (binary/regression/multiclass) to make it easier to review/update. Note: I've updated the JIRA to: https://issues.apache.org/jira/browse/SPARK-24101 Which is a child of JIRA: https://issues.apache.org/jira/browse/SPARK-18693 ## How was this patch tested? I added tests to the metrics class. Closes#17086 from imatiach-msft/ilmat/multiclass-evaluate. Authored-by: Ilya Matiach <ilmat@microsoft.com> Signed-off-by: Sean Owen <sean.owen@databricks.com>

…ed weight column for regression evaluator ## What changes were proposed in this pull request? The evaluators BinaryClassificationEvaluator, RegressionEvaluator, and MulticlassClassificationEvaluator and the corresponding metrics classes BinaryClassificationMetrics, RegressionMetrics and MulticlassMetrics should use sample weight data. I've closed the PR: apache#16557 as recommended in favor of creating three pull requests, one for each of the evaluators (binary/regression/multiclass) to make it easier to review/update. The updates to the regression metrics were based on (and updated with new changes based on comments): https://issues.apache.org/jira/browse/SPARK-11520 ("RegressionMetrics should support instance weights") but the pull request was closed as the changes were never checked in. ## How was this patch tested? I added tests to the metrics class. Closesapache#17085 from imatiach-msft/ilmat/regression-evaluate. Authored-by: Ilya Matiach <ilmat@microsoft.com> Signed-off-by: Sean Owen <sean.owen@databricks.com>

…ed weight column for multiclass classification evaluator ## What changes were proposed in this pull request? The evaluators BinaryClassificationEvaluator, RegressionEvaluator, and MulticlassClassificationEvaluator and the corresponding metrics classes BinaryClassificationMetrics, RegressionMetrics and MulticlassMetrics should use sample weight data. I've closed the PR: apache#16557 as recommended in favor of creating three pull requests, one for each of the evaluators (binary/regression/multiclass) to make it easier to review/update. Note: I've updated the JIRA to: https://issues.apache.org/jira/browse/SPARK-24101 Which is a child of JIRA: https://issues.apache.org/jira/browse/SPARK-18693 ## How was this patch tested? I added tests to the metrics class. Closesapache#17086 from imatiach-msft/ilmat/multiclass-evaluate. Authored-by: Ilya Matiach <ilmat@microsoft.com> Signed-off-by: Sean Owen <sean.owen@databricks.com>

…ed weight column for regression evaluator ## What changes were proposed in this pull request? The evaluators BinaryClassificationEvaluator, RegressionEvaluator, and MulticlassClassificationEvaluator and the corresponding metrics classes BinaryClassificationMetrics, RegressionMetrics and MulticlassMetrics should use sample weight data. I've closed the PR: apache#16557 as recommended in favor of creating three pull requests, one for each of the evaluators (binary/regression/multiclass) to make it easier to review/update. The updates to the regression metrics were based on (and updated with new changes based on comments): https://issues.apache.org/jira/browse/SPARK-11520 ("RegressionMetrics should support instance weights") but the pull request was closed as the changes were never checked in. ## How was this patch tested? I added tests to the metrics class. Closesapache#17085 from imatiach-msft/ilmat/regression-evaluate. Authored-by: Ilya Matiach <ilmat@microsoft.com> Signed-off-by: Sean Owen <sean.owen@databricks.com>

…ed weight column for binary classification evaluator ## What changes were proposed in this pull request? The evaluators BinaryClassificationEvaluator, RegressionEvaluator, and MulticlassClassificationEvaluator and the corresponding metrics classes BinaryClassificationMetrics, RegressionMetrics and MulticlassMetrics should use sample weight data. I've closed the PR: #16557 as recommended in favor of creating three pull requests, one for each of the evaluators (binary/regression/multiclass) to make it easier to review/update. ## How was this patch tested? I added tests to the metrics and evaluators classes. Closes#17084 from imatiach-msft/ilmat/binary-evalute. Authored-by: Ilya Matiach <ilmat@microsoft.com> Signed-off-by: Sean Owen <sean.owen@databricks.com>

imatiach-msft force-pushed the ilmat/evaluate-with-weights branch from 397c26b to 808ca6bCompare January 13, 2017 15:55

imatiach-msft changed the title ~~[SPARK-18693][ML][MLLIB][WIP] ML Evaluators should use weight column~~[SPARK-18693][ML][MLLIB] ML Evaluators should use weight columnJan 13, 2017

imatiach-msft force-pushed the ilmat/evaluate-with-weights branch from 808ca6b to 228bbfbCompare January 17, 2017 18:54

imatiach-msft force-pushed the ilmat/evaluate-with-weights branch from e2873f2 to d589dfeCompare January 18, 2017 15:39

imatiach-msft force-pushed the ilmat/evaluate-with-weights branch from c1f5f09 to ba68f72Compare January 23, 2017 16:28

imatiach-msft added 4 commits February 27, 2017 12:57

[SPARK-18693][ML] Evaluators should use weight column
fb8f7fd

Updated comment
7d40160

Fixed merge with latest
f0a9c54

Updated to latest code
a0fc4c3

imatiach-msft force-pushed the ilmat/evaluate-with-weights branch from 6dbb0ad to a0fc4c3Compare February 27, 2017 18:03

imatiach-msft closed this Feb 27, 2017

Uh oh!

Conversation

imatiach-msft commented Jan 12, 2017

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Jan 12, 2017

Uh oh!

SparkQA commented Jan 12, 2017

Uh oh!

imatiach-msft commented Jan 12, 2017

Uh oh!

imatiach-msft commented Jan 12, 2017

Uh oh!

imatiach-msft commented Jan 12, 2017

Uh oh!

SparkQA commented Jan 12, 2017

Uh oh!

SparkQA commented Jan 13, 2017

Uh oh!

SparkQA commented Jan 13, 2017

Uh oh!

imatiach-msft commented Jan 13, 2017

Uh oh!

imatiach-msft commented Jan 13, 2017

Uh oh!

imatiach-msft commented Jan 13, 2017

Uh oh!

imatiach-msft commented Jan 13, 2017

Uh oh!

imatiach-msft commented Jan 13, 2017

Uh oh!

imatiach-msft commented Jan 13, 2017

Uh oh!

SparkQA commented Jan 13, 2017

Uh oh!

SparkQA commented Jan 13, 2017

Uh oh!

imatiach-msft commented Jan 17, 2017

Uh oh!

SparkQA commented Jan 17, 2017

Uh oh!

SparkQA commented Jan 17, 2017

Uh oh!

imatiach-msft commented Jan 18, 2017

Uh oh!

SparkQA commented Jan 18, 2017

Uh oh!

imatiach-msft commented Jan 18, 2017

Uh oh!

imatiach-msft commented Jan 18, 2017

Uh oh!

SparkQA commented Jan 18, 2017

Uh oh!

SparkQA commented Jan 18, 2017

Uh oh!

SparkQA commented Jan 19, 2017

Uh oh!

imatiach-msft commented Jan 23, 2017

Uh oh!

imatiach-msft commented Jan 23, 2017

Uh oh!

SparkQA commented Jan 23, 2017

Uh oh!

SparkQA commented Jan 23, 2017

Uh oh!

imatiach-msft commented Jan 25, 2017

Uh oh!

imatiach-msft commented Jan 26, 2017

Uh oh!

imatiach-msft commented Jan 26, 2017

Uh oh!

srowen commented Jan 26, 2017

Uh oh!

sethah commented Jan 26, 2017

Uh oh!

thunterdb commented Feb 17, 2017

Uh oh!

imatiach-msft commented Feb 27, 2017

Uh oh!