[SPARK-16934][ML][MLLib]Update LogisticCostAggregator serialization code to make it consistent with LinearRegression

[WeichenXu123]

What changes were proposed in this pull request?

Update LogisticCostAggregator serialization code to make it consistent with #14109

How was this patch tested?

MLlib 2.0:

After this PR:

[WeichenXu123]

cc @sethah

[srowen]

Let's put this into #14109

[WeichenXu123]

Oh..its another algorithm and there are several different details so in order to make it clear I create a separated PR to discuss it , thanks!

[SparkQA]

Test build #63315 has finished for PR 14520 at commit 417aa1e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MLnick]

@WeichenXu123 I believe #13729 already took care of the actual serialization issue. Out of interest have you tested this impl here for a difference in shuffle data read/write?

However, #14109 and now #14519 do take a slightly different approach with BC vars and transient, so it is probably worthwhile to make them all consistent as per #14109 (comment).

cc @sethah@yanboliang@dbtsai

[sethah]

Well, I suppose this won't go into another PR since the other one got merged. I think it's correct to make this match the approach taken in Linear Regression. The current code doesn't quite match though, so could you take a look at #14109 and make this line up. Also, I'd like to see a comparison of MLlib and this patch to verify that the shuffle write size is the same for the tasks, to make sure we haven't undone anything. It's a good sanity check.

[WeichenXu123]

@MLnick
The main improvement here is about localFeaturesStd,
in previous code, each call on CostFun.calculate will do a serialization and broadcast on vector.
mark localFeaturesStd as spark broadcast var will avoid this problem.
Thanks!

[sethah]

The change here does not really affect serialization. Spark automatically broadcasts the coefficients each time calculate is called before, and marking it as a broadcast variable explicitly won't likely have much of a performance effect (based on my own testing and the description here). What we need to do here is to change the structure of the aggregator to match up with the fix for LeastSquaresAggregator by passing the featuresStd and coefficients as constructor args, but marking them as @transient lazy val.

I'm in favor of explicitly broadcasting the coefficients too, as was done in LeastSquaresAggregator, but we should explicitly destroy them as well. Thanks for working on this!

[WeichenXu123]

@sethah
Thanks for your careful review!
The PR here already passing the bcFeaturesStd and bcCoeffs as constructor args to the LogisticAggregator, like your PR #14109

You mean add another two member into LogisticAggregator like
@transient lazy val featureStd = bcFeatureStd.value
@transient lazy val coeffs = bcCoeff.value
?

And explicitly destroy broadcast I will add it soon!
Thanks.

[WeichenXu123]

cc @yanboliang Thanks!

[SparkQA]

Test build #63719 has finished for PR 14520 at commit 90b981f.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #63720 has finished for PR 14520 at commit 04baa3c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #63722 has finished for PR 14520 at commit 2b6c867.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yanboliang]

We should explicitly destroy bcCoeffs at the end of calculate by bcCoeffs.destroy(blocking = false) for each iteration.

[WeichenXu123]

@sethah I attach the test result and it looks good.

[WeichenXu123]

@yanboliang Thanks for carefully review!

[SparkQA]

Test build #63744 has finished for PR 14520 at commit 87f5417.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dbtsai]

Call it bcCoefficients for consistency with other changes.

[MLnick]

@WeichenXu123 would you mind updating the title of the JIRA and PR, as well as the description, to reflect the fact that this is not actually affecting serialization, but more to update the approach to be consistent with the other changes made in #14109?

This is just for record-keeping to avoid any confusion in future. Thanks!

[SparkQA]

Test build #63778 has finished for PR 14520 at commit e7ff240.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #63779 has finished for PR 14520 at commit efe3d38.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

@yanboliang are you OK with this?

[yanboliang]

LGTM, merged into master. Thanks!