[WIP] [SPARK-1328] Add vector statistics

[yinxusen]

As with the new vector system in MLlib, we find that it is good to add some new APIs to precess the RDD[Vector]. Beside, the former implementation of computeStat is not stable which could loss precision, and has the possibility to cause Nan in scientific computing, just as said in the SPARK-1328.

APIs contain:

• rowMeans(): RDD[Double]
• rowNorm2(): RDD[Double]
• rowSDs(): RDD[Double]
• colMeans(): Vector
• colMeans(size: Int): Vector
• colNorm2(): Vector
• colNorm2(size: Int): Vector
• colSDs(): Vector
• colSDs(size: Int): Vector
• maxOption((Vector, Vector) => Boolean): Option[Vector]
• minOption((Vector, Vector) => Boolean): Option[Vector]
• rowShrink(): RDD[Vector]
• colShrink(): RDD[Vector]

This is working in process now, and some more APIs will add to LabeledPoint. Moreover, the implicit declaration will move from MLUtils to MLContext later.

[AmplabJenkins]

Merged build triggered. Build is starting -or- tests failed to complete.

[AmplabJenkins]

Merged build started. Build is starting -or- tests failed to complete.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13565/

[mengxr]

@yinxusen Thanks for working on this! I don't think row statistics are important because they represent values for different features. For column statistics, instead of implementing each statistic separately, we can compute all common statistics like (n, nnz, mean, variance, max, min) in a single job. This adds little overhead to the computation.

Btw, Vector.toArray is an expensive operation for SparseVector. You should use breeze's axpy to aggregate vectors.

[yinxusen]

Yes, @mengxr I find it is cool to merge them together and rewrite the SD in a single pass, just like the page said. I'll fix it ASAP.

[AmplabJenkins]

Build triggered. Build is starting -or- tests failed to complete.

[AmplabJenkins]

Build started. Build is starting -or- tests failed to complete.

[yinxusen]

I remove all APIs and reserve a new one call statistics(), which can compute six statistical quantities in a single pass.

Row-wise statistical quantities are removed because they are meaningless.

Column-wise shrinkage and row-wise shrinkage are also removed, because they belong to feature/data selection. We should re-consider them and write in other semantic class such as LabeledPoint.

[AmplabJenkins]

Build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13585/

[mengxr]

Please wrap the return type to a class called VectorRDDStatisticalSummary that provides mean, variance, count: Long, min, max, and std. So later we can add more summary statistics to it. I would also recommend changing the method name to summarize or summarizeStatistics.

[mengxr]

@yinxusen I want to see whether we can improve the performance. First of all, the complexity should be O(nnz) instead of O(n d). This is a little bit tricky. Basically, we don't need to update zeros for a sparse vector. Suppose we have the following column:

1.0
0.0
2.0
0.0
3.0
0.0
0.0

To apply the formula for mean/variance, you need to do the update at the second row. This is actually not necessary because the summary statistics are invariant to the ordering. Imagine we have this column re-arranged:

1.0
2.0
3.0
0.0
0.0
0.0
0.0

which still have the same summary statistics. We can skip all zeros and do mean/variance updates based on the non-zero count instead of global count. After aggregate, we know we need to accumulate 4 zeros to the column, which is a constant-time operation.

I'm okay if you don't have enough time to make the change in this PR. Put a TODO and we can fix it later.

[yinxusen]

@mengxr I am not very sure of the concept of sparse vector. In your example, do you mean the column is Vector(1.0, 0.0, 2.0, 0.0, 3.0, 0.0, 0.0) or
RDD( Vector(1.0), Vector(0.0), Vector(2.0), Vector(0.0), Vector(3.0), Vector(0.0), Vector(0.0) )?

If it is the case 1, then it is easy to rewrite it in O(nnz), otherwise, it will be difficult, because we cannot judge whether a column is sparse or not before we count the nnz. If the case 1 is your mean, then I think I should treat sparse vector different with the dense one with the following code:

RDD.take(1).head.type match { case DenseVector[Double] => xxx case SparseVector[Double] => xxx }.

[mengxr]

@yinxusen I mean a column. You don't need to check the type as you already use activeIterator to get the non-zero elements. The approach I suggested is to only update mean/variance/min/max for non-zeros. After we scan all the items, we know how many zeros we need to append to each column, and then we do a constant-time update for each column.

[yinxusen]

@mengxr Ah... I totally understand your mean. Code is on the way.

[AmplabJenkins]

Build triggered.

[AmplabJenkins]

Build started.

[AmplabJenkins]

Build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13630/

[yinxusen]

Build failed might be caught by this empty line in the end. I'll fix it in next commit with other problems.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14025/

[yinxusen]

Conflict in MLUtils and RowMatrix. I think it is OK now.
2014-4-11 AM5:31于 "Matei Zaharia" notifications@github.com写道：

Hey, unfortunately this no longer merges cleanly. Do you mind rebasing it?
I think some conflicting changes happened in MLUtils.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/268#issuecomment-40145491
.

[pwendell]

Thanks, merged!