[SPARK-16495] [MLlib]Add ADMM optimizer in mllib package

[ZunwenYou]

Alternating Direction Method of Multipliers (ADMM) is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas.
Details can be found in the S. Boyd's paper.

JIRA Issue: https://issues.apache.org/jira/browse/SPARK-16495

[AmplabJenkins]

Can one of the admins verify this patch?

[ZunwenYou]

@MLnick please have a look at this.

[sethah]

@ZunwenYou Would you mind addressing the comments in the JIRA first? Adding a new optimization algorithm to an API that is now deprecated definitely warrants more high level discussion before code review should proceed, IMO.

[MLnick]

@ZunwenYou sorry if I was not clear on the JIRA. I said there that this should probably be done as a Spark package external to the core initially. That way you can gather some user feedback and performance numbers etc.

If this is to be implemented within Spark then as Seth mentions it makes more sense to go into the ml API. In this case we need more work and discussion to see how to add different optimizers and make them available to the different models with a unified API.

It would also need some performance comparisons to the current optimizer options.

[ZunwenYou]

@MLnick You are right. We have apply ADMM to Sparse Logistic Regression with L1 norm in some CTR applications, the data sets of these applications almost consist of 10 million dimension and 100 million samples. Actually, ADMM's LR is fast than SGD on large scale data.

I can provide some performance comparisons to SGD or OWLQN on our data set，but I wonder if the performance comparisons are convinced because our data set is private. Do you have any idea about data set for preformance comparisons?

[MLnick]

I'd recommend (a) generate some data; and/or (b) take a look at some larger public datasets (or samples thereof) such as Criteo (https://www.kaggle.com/c/criteo-display-ad-challenge/data) or Avito (https://www.kaggle.com/c/avito-context-ad-clicks/data)

[debasish83]

ADMM is already available as a breeze solver (BFGS, OWLQN, NonlinearMinimizer) which is integrated with ml/mllib...It will be great if you can look into it and let me know if you need pointers in running comparisons with OWLQN:
https://github.com/scalanlp/breeze/blob/master/math/src/main/scala/breeze/optimize/proximal/NonlinearMinimizer.scala
This is implemented based on the paper you cited.