[SPARK-4259][MLlib]: Add Power Iteration Clustering Algorithm with Gaussian Similarity Function

[fjiang6]

Add single pseudo-eigenvector PIC
Including documentations and updated pom.xml with the following codes:
mllib/src/main/scala/org/apache/spark/mllib/clustering/PIClustering.scala
mllib/src/test/scala/org/apache/spark/mllib/clustering/PIClusteringSuite.scala

[SparkQA]

Test build #26252 has started for PR 4254 at commit 121e4d5.

• This patch merges cleanly.

[SparkQA]

Test build #26252 has finished for PR 4254 at commit 121e4d5.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26252/
Test FAILed.

[mengxr]

Is it possible to do Gaussian similarity in another PR? It should be part of the feature transformation but not within PIC. It would be easier for code review if the PR is minimal.

[mengxr]

Let's not handle I/O here. We can have an example code under examples/ and load files there.

[javadba]

OK

[mengxr]

You don't need ordering vt before calling kmeans.

[javadba]

true, it was used sorted for other purpose, will remove.

[mengxr]

@fjiang6@javadba Please focus on the public APIs first and then the implementation. The best way to check public APIs is generating the html doc and look what are exposed to users. We can probably refactor the test code in later PRs.

[SparkQA]

Test build #26354 has started for PR 4254 at commit 24fbf52.

• This patch merges cleanly.

[mengxr]

This is too large for unit tests. Unit tests should be as minimal as possible. For this one, we can construct a very small graph, compute its eigenvector, and derive the clustering result manually, then verify PIC result. For example

a - b - c - g - h
| \ | | \ |
d - e - f i - j

Assign each edge distance 1 and run PIC with k = 2. The solution should be clear.

[javadba]

OK

[SparkQA]

Test build #26354 has finished for PR 4254 at commit 24fbf52.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class PowerIterationClustering(

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26354/
Test PASSed.

[SparkQA]

Test build #26420 has started for PR 4254 at commit f292f31.

• This patch merges cleanly.

[SparkQA]

Test build #26423 has started for PR 4254 at commit 4550850.

• This patch merges cleanly.

[SparkQA]

Test build #26420 has finished for PR 4254 at commit f292f31.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class PowerIterationClusteringModel(

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26420/
Test PASSed.

[SparkQA]

Test build #26423 has finished for PR 4254 at commit 4550850.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class PowerIterationClusteringModel(

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26423/
Test PASSed.

[mengxr]

Should use relative path "api/graphx/...". See examples in this markdown file.

[mengxr]

LGTM except minor user guide issues, which will be addressed in SPARK-5503. I've merged this into master. Thanks for the contributing! (Now MLlib depends on GraphX.)