[SPARK-11520][ML] RegressionMetrics should support instance weights

[Lewuathe]

This will be important to improve LinearRegressionSummary, which currently has a mix of weighted and unweighted metrics.

[SparkQA]

Test build #46525 has finished for PR 9907 at commit 1d4a5fd.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[Lewuathe]

@mengxr@jkbradley Could you review this? Thanks.

[thunterdb]

We should rename this summary variable, it is used 3 times for different objects.

Also, sample is a bit too generic here, and using the _xx methods are not very intuitive. I suggest you unpack the tuple fully: { case (currentSummary, (vec, weight)) => currentSummary.add(vec, weight) }

[thunterdb]

@Lewuathe thanks for your patch. I think it will require more work in RegressionMetrics to fully implement weighted metrics. We need to do the following changes:

• expose weightSum in MultivariateStatisticalSummary (as a developer API)
• the computations of SSreg and SStot should take the weights into account
• in RegressionMetrics, all references to summary.count should be replaced by summary.weightSum
  Given that the default weights are 0, it should give the same result.