[SPARK-10524][ML] Use the soft prediction to order categories' bins

[viirya]

JIRA: https://issues.apache.org/jira/browse/SPARK-10524

Currently we use the hard prediction (ImpurityCalculator.predict) to order categories' bins. But we should use the soft prediction.

[SparkQA]

Test build #42382 has finished for PR 8734 at commit 84260ca.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jkbradley]

Can you please update this test to call binsToBestSplit directly? You can change it to be private[tree] so that it's callable from this test suite.

[jkbradley]

Ping

[viirya]

In order to call binsToBestSplit directly, we need to expose many details of findBestSplits too, e.g., binSeqOp, getNodeToFeatures and partitionAggregates...etc., because binsToBestSplit needs binAggregates and featuresForNode..etc. as parameters. Is it a good idea?

[jkbradley]

I'll have bandwidth to get this merged now, so I'll watch for updates. Thanks!

[jkbradley]

Ping! Please let me know if you don't have time to work on this, and I can take it over. Thanks

[viirya]

@jkbradley Sorry for replying late. I will try to finish this soon. Thanks.

[jkbradley]

OK thanks!

[SparkQA]

Test build #49757 has finished for PR 8734 at commit 2c32350.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sethah]

I don't believe this is correct. Ordering by the probability of the prediction is essentially the same as ordering by impurity. That's because when the impurity is low, the predicted value will have high probability and vice versa.

From Hastie, Tibshirani, and Friedman:
"We order the predictor classes according to the proportion falling in outcome class 1. Then we split this predictor as if it were an ordered predictor."

For binary category I think it should be as @jkbradley suggested categoryStats.stats(1)

[viirya]

As I saw from the implementation, categoryStats.stats(1) is just the count of class 1, not the proportion falling in outcome class 1. Are we going to order bins by that?

[sethah]

Finding the proportion falling in outcome class 1 simply requires division of the counts by a constant. Since we're just using that number for an ordering, constant division won't matter. They are the same.

My initial comment has a typo. It should say for a "binary outcome", not "binary category".

[viirya]

Yeah, I see. I was thinking we are going to order them by soft prediction of each bin. Actually what we want is to order them by soft prediction of certain class.

[SparkQA]

Test build #49865 has finished for PR 8734 at commit cd25214.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #49874 has finished for PR 8734 at commit a37d3d8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sethah]

I think you meant categoryStats.stats.length == 2. categoryStats.count is the count of data points falling into that particular bin. Since we are trying to determine here whether this is regression or binary classification, I think checking if (binAggregates.metadata.isClassification) is more clear.

Additionally, the code under the if and else statements of centroidForCategories is identical except for a single line. It seems cleaner to restructure to something like:

valcentroidForCategories=Range(0, numCategories).map { case featureValue =>valcategoryStats=
binAggregates.getImpurityCalculator(nodeFeatureOffset, featureValue)
valcentroid=if (categoryStats.count !=0) {
if (binAggregates.metadata.isMulticlass) {
// multiclass classification
categoryStats.calculate()
} elseif (binAggregates.metadata.isClassification) {
// binary classification
categoryStats.stats(1)
} else {
// regression
categoryStats.predict
}
} else {
Double.MaxValue
}
(featureValue, centroid)
}

[viirya]

@sethah Thanks. You are right. I didn't read this part of codes thoroughly.

[SparkQA]

Test build #49934 has finished for PR 8734 at commit 5c44e23.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jkbradley]

@viirya Thanks for the updates.

I think the code is correct now, but I'm going to send you a PR (to update this PR) in order to improve the test. I agree with @sethah that the current test does not really test anything.

@sethah Does it look good to you, other than the test?

[sethah]

Yes, LGTM pending the improved test, thanks!

[jkbradley]

Here it is: [https://github.com/viirya/spark-1/pull/1]

[viirya]

@jkbradley Great thanks. I've merged your PR.

[SparkQA]

Test build #51012 has finished for PR 8734 at commit 2bbe037.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jkbradley]

LGTM! Thanks @viirya and @sethah

I'll merge with master and see how far back I can backport it easily.