[MLlib] [SPARK-2885] DIMSUM: All-pairs similarity

[rezazadeh]

All-pairs similarity via DIMSUM

Compute all pairs of similar vectors using brute force approach, and also DIMSUM sampling approach.

Laying down some notation: we are looking for all pairs of similar columns in an m x n RowMatrix whose entries are denoted a_ij, with the i’th row denoted r_i and the j’th column denoted c_j. There is an oversampling parameter labeled ɣ that should be set to 4 log(n)/s to get provably correct results (with high probability), where s is the similarity threshold.

The algorithm is stated with a Map and Reduce, with proofs of correctness and efficiency in published papers [1] [2]. The reducer is simply the summation reducer. The mapper is more interesting, and is also the heart of the scheme. As an exercise, you should try to see why in expectation, the map-reduce below outputs cosine similarities.

[1] Bosagh-Zadeh, Reza and Carlsson, Gunnar (2013), Dimension Independent Matrix Square using MapReduce, arXiv:1304.1467 http://arxiv.org/abs/1304.1467

[2] Bosagh-Zadeh, Reza and Goel, Ashish (2012), Dimension Independent Similarity Computation, arXiv:1206.2082 http://arxiv.org/abs/1206.2082

Testing

Tests for all invocations included.

Added L1 and L2 norm computation to MultivariateStatisticalSummary since it was needed. Added tests for both of them.

[SparkQA]

QA tests have started for PR 1778. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17926/consoleFull

[SparkQA]

QA results for PR 1778:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17926/consoleFull

[SparkQA]

QA tests have started for PR 1778. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17929/consoleFull

[SparkQA]

QA results for PR 1778:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17929/consoleFull

[rezazadeh]

The binary backwards compatibility check doesn't like adding a new method to the trait MultivariateStatisticalSummary. Suggestions on binary compatibility welcome, @mengxr

[srowen]

As a meta-question, what's the theory about what implementations should go into Spark, and which should be external? Not everything needs to be in a "core" library like MLlib. I know Mahout suffered mightily from adding a lot of implementations without much regard to their use or support. I'm not suggesting anything either way about this algorithm. If there's a working theory about what's in and out of scope, I'd love to see it and maybe make sure that people don't implement things for contribution that are too niche.

[rezazadeh]

Having all-pairs similarity in spark has been requested several times. e.g. http://bit.ly/XAFGs8 , and also by @freeman-lab . This algorithm is also a part of Scalding: twitter/scalding#833

[pwendell]

@rezazadeh mind putting [MLlib] in the title here? That way it gets sorted correctly by our internal reivew tools.

[freeman-lab]

@srowen agreed the core vs external library question is important. The requirements here seem reasonable, but there's still gray area. For example, we have lots of analyses that are known / accepted but should I think remain external because they are for specific data types (images & time series). Re: this particular algorithm, it's definitely something we're interested in using, sounds like others are too.

[mengxr]

@rezazadeh Do you mind creating a JIRA for this and then add [SPARK-####] to the title? We also want to learn more about the theory, especially the relation between storage/computation complexity and failure rate.

Btw, to me, finding similar rows (observations) is more natural than finding similar columns.

[SparkQA]

QA tests have started for PR 1778. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18061/consoleFull

[rezazadeh]

@mengxr Updated the PR to compute column magnitude as a method in RowMatrix so that binary compatibility shouldn't be a problem. This allowed me to use breeze too, which should take advantage of hardware acceleration when possible.

@srowen@freeman-lab@mengxr I added a JIRA for this PR, and clearly laid out why it is worthwhile adding this functionality to MLlib. https://issues.apache.org/jira/browse/SPARK-2885

[SparkQA]

QA results for PR 1778:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18061/consoleFull

[SparkQA]

QA tests have started for PR 1778. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18076/consoleFull

[SparkQA]

QA results for PR 1778:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18076/consoleFull

[SparkQA]

QA tests have started for PR 1778 at commit aea0247.

• This patch does not merge cleanly!

[SparkQA]

QA tests have started for PR 1778 at commit 3467cff.

• This patch merges cleanly.

[rezazadeh]

@mengxr I also added broadcasting of p and v to further optimize space usage. Also now we're avoiding divide by zero if there is a column with zero magnitude.

[SparkQA]

QA tests have finished for PR 1778 at commit aea0247.

• This patch fails unit tests.
• This patch does not merge cleanly!

[SparkQA]

QA tests have finished for PR 1778 at commit 3467cff.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

test this please

[SparkQA]

QA tests have started for PR 1778 at commit ee8bd65.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 1778 at commit ee8bd65.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rezazadeh]

Only the binary compatibility test is failing, which is expected.

[mengxr]

@rezazadeh Could you set the exclusion rules in dev/MimaExcludes.scala?

[SparkQA]

QA tests have started for PR 1778 at commit 4eb71c6.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 1778 at commit 4eb71c6.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rezazadeh]

Jenkins, test this please.

[SparkQA]

QA tests have started for PR 1778 at commit 4eb71c6.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 1778 at commit 4eb71c6.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

QA tests have started for PR 1778 at commit 404c64c.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 1778 at commit 404c64c.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

LGTM. Merged into master! Thanks @rezazadeh !

[rezazadeh]

Thanks for the review @mengxr !

[appierys]

Does anyone know how to extend this to the 'Cross Product' case as mentioned in the paper?