[SPARK-11496][GRAPHX] Parallel implementation of personalized pagerank

[moustaki]

This implements a parallel version of personalized pagerank, which runs all propagations for a list of source vertices in parallel.

I ran a few benchmarks on the full DBpedia graph. When running personalized pagerank for only one source node, the existing implementation is twice as fast as the parallel one (because of the SparseVector overhead). However for 10 source nodes, the parallel implementation is four times as fast. When increasing the number of source nodes, this difference becomes even greater.

[dbtsai]

Jenkins, test this please

[SparkQA]

Test build #44990 has finished for PR 9457 at commit f41975e.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[moustaki]

Jenkins, test this please

[dbtsai]

Jenkins, add to whitelist

[SparkQA]

Test build #45033 has finished for PR 9457 at commit 3605e40.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dbtsai]

Since mllib depneds on graphx, please remove the breeze dependencies in mllib.

[dbtsai]

Is return type Graph[SparseVector[Double], Double] required in this algorithm? Typically, we don't return breeze type in the public api. However, mllib's sparse vector is in mllib package, so there is no way to have graphx depending on mllib which will cause circular dependencies.

I'm working on refactoring out the basic type in mllib into separate package, but this will not happen soon. (will likely happen in 1.7).

[moustaki]

Thanks for all the comments @dbtsai. I'll go through your various comments soon. About your last one, what would you recommend for the time being?

[dbtsai]

@moustaki For the sparse vector issue, we can wait for the change in spark 1.7 since this PR will not be in 1.6. Thanks.

[moustaki]

@dbtsai Just went through all your comments. Thanks a lot for the feedback!

[SparkQA]

Test build #46574 has finished for PR 9457 at commit 8506353.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dbtsai]

@moustaki This PR looks good to me. I'll start to work on refactoring the mllib vector out after 1.6 release, so graphx can use those data structure. Then we can change BSV to mllib sparse vector implementation. Thanks.

[moustaki]

@dbtsai Sounds good - do you want to hold off merging in the meantime?

[dbtsai]

@moustaki yes, we need to hold off now until 1.7 window.

[rxin]

Thanks for the pull request. I'm going through a list of pull requests to cut them down since the sheer number is breaking some of the tooling we have. Due to lack of activity on this pull request, I'm going to push a commit to close it. Feel free to reopen it or create a new one.

[moustaki]

Sounds good. @dbtsai Let me know when your mllib vector refactor is out and I'll re-open this PR.

[dbtsai]

Hello @moustaki ,

The work of having standalone vectors and matrices had been done in SPARK-13944, and what you need to do is adding the dependency of mllib-local into graphx with the following example https://github.com/apache/spark/pull/12298/files to use those types in graphx.

Thanks.

[moustaki]

Thanks @dbtsai!

[makalaaneesh]

@moustaki Hey, I am currently working on a project that requires to perform PersonalizedPageRanks in parallel. Has this been merged, cause I can't find official documentation on this.

[moustaki]

@makalaaneesh Yes, it has been superseeded by another PR #9457 and merged. It's available in 2.1.