[SPARK-8468][ML] Take the negative of some metrics in RegressionEvaluator to get correct cross validation

[viirya]

JIRA: https://issues.apache.org/jira/browse/SPARK-8468

[SparkQA]

Test build #35266 has finished for PR 6905 at commit b5f52c1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

You would at least have to change the docs in Evaluator to say that evaluate no longer necessarily returns larger values for better evaluations.

I don't see that you changed RegressionEvaluator to flip the logic though?

There's more of a problem with that though. 3 of the 4 metrics in RegressionEvaluator are worse when larger: RMSE, MSE, MAE. But R^2 is not. So this would still have a similar bug.

Another possibility is to invert the result of RMSE, MSE, MAE. For eval purposes, their relative ranking is all that matters so returning 1/x as the evaluation criteria is fine, for example. That would let you fully fix this without any API change.

[viirya]

Thanks. I didn't notice there is R^2 that is needed to maximize. The simplest solution should be to invert the result of RMSE, MSE, MAE.

[SparkQA]

Test build #35278 has finished for PR 6905 at commit c3dd8d9.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

@srowen@viirya I think using - would be better than 1 /, at least it is easier to see the absolute value.

[viirya]

@mengxr agreed. Updated.

[mengxr]

LGTM pending Jenkins. Thanks for the quick fix and @chelseaz for reporting the bug!

[chelseaz]

Thanks for the fix @viirya!

[srowen]

Arguing with myself: I see the value of the eval is used in some log statements including its mean. This may be a little less than useful when it is some inverse RMSE but still probably a decent solution

[SparkQA]

Test build #35285 has finished for PR 6905 at commit 16e3b2c.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

test this please

[SparkQA]

Test build #35311 has finished for PR 6905 at commit 16e3b2c.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

@viirya Could you fix the python unit tests?

**********************************************************************
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/ml/evaluation.py", line 162, in __main__.RegressionEvaluator
Failed example:
evaluator.evaluate(dataset)
Expected:
2.842...
Got:
-2.8424854081906372
**********************************************************************
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/ml/evaluation.py", line 166, in __main__.RegressionEvaluator
Failed example:
evaluator.evaluate(dataset, {evaluator.metricName: "mae"})
Expected:
2.649...
Got:
-2.6496454299999996
**********************************************************************

[jkbradley]

@viirya Could you also please document the meaning of the various metrics in RegressionEvaluator? Users will be surprised that those values are negated. The doc can go in the "metricName" Param Scala/Python doc. Thanks!

[SparkQA]

Test build #35351 has finished for PR 6905 at commit 930d3db.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class StreamingKMeansModel(KMeansModel):
  • class StreamingKMeans(object):

[jkbradley]

LGTM merging into branch-1.4 and master
Thanks!