Changes to support KMeans with large feature space

[levin-royl]

The problem:

In Spark's KMeans code the center vectors are always represented as dense vectors. As a result, when each such center has a large domain space the algorithm quickly runs out of memory. In my example I have a feature space of around 50000 and k ~= 500. This sums up to around 200MB RAM for the center vectors alone while in fact the center vectors are very sparse and require a lot less RAM.
Since I am running on a system with relatively low resources I keep getting OutOfMemory errors. In my setting it is OK to trade off runtime for using less RAM. This is what I set out to do in my solution while allowing users the flexibility to choose.

My solution:

Allow the kmeans algorithm to accept a VectorFactory which decides when vectors used inside the algorithm should be sparse and when they should be dense. For backward compatibility the default behavior is to always make them dense (like the situation is now). But now potentially the user can provide a SmartVectorFactory (or some proprietary VectorFactory) which can decide to make vectors sparse.

For this I made the following changes:
(1) Added a method called reassign to SparseVectors allowing to change the indices and values
(2) Allow axpy to accept SparseVectors
(3) create a trait called VectorFactory and two implementations for it that are used within KMeans code

[yinxusen]

Hi @levin-royl, you need to remove the two log files and create a JIRA. See https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark

[levin-royl]

Thank you, I removed the log files and added the following JIRA request:

https://issues.apache.org/jira/browse/KYLIN-1326

[yinxusen]

You may create your JIRA in a wrong place. Not Kylin, but Spark https://issues.apache.org/jira/browse/SPARK

[levin-royl]

Sorry, I am a little new to this. For some reason when choosing "create new" I only had the options: Kylin, Atlas or Apache Infrastructure. Now through the link you sent I created the following JIRA request in Spark:

https://issues.apache.org/jira/browse/SPARK-12861

[levin-royl]

Hi, just wanted to know if there are any unhanded items on my end WRT this change. Thanks.

[srowen]

Read https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark first. You may want to search for duplicate JIRAs too. There are several on this topic and k-means.

[levin-royl]

Hi, there are indeed some similar issues I found, e.g.:

https://issues.apache.org/jira/browse/SPARK-4039
https://issues.apache.org/jira/browse/SPARK-1212
mesos/spark#736

But the difference is that in the problem I describe reducing the dimensions of the problem (i.e., the feature space) to allow using dense vectors is not suitable. Also, the solution I implemented supports this while allowing full flexibility to the user --- i.e., using the default dense vector implementation or selecting an alternative (only when the default it is not desired).

I will update the JIRA issue on this as well.

Please advise if there are any additional steps I need to do at this point.

Thanks in advance.

[levin-royl]

Hi, there are indeed some similar issues I found, e.g.:

https://issues.apache.org/jira/browse/SPARK-4039
https://issues.apache.org/jira/browse/SPARK-1212
mesos/spark#736

But the difference is that in the problem I describe reducing the dimensions of the problem (i.e., the feature space) to allow using dense vectors is not suitable. Also, the solution I implemented supports this while allowing full flexibility to the user --- i.e., using the default dense vector implementation or selecting an alternative (only when the default it is not desired).

I will update the JIRA issue on this as well.

Please advise if there are any additional steps I need to do at this point.

Thanks in advance.

[levin-royl]

I wanted to know if you took a look at the code and the proposed solution in general. Are there any comments?

Thanks.

[thunterdb]

@levin-royl it looks like @hhbyyh has a branch with some code that tackles the same issue (see the jira discussion for more information), you may want to coordinate there.

Also, I suggest you take again a look at the pull request section of the guidelines https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark

[hhbyyh]

I didn't send a PR because there's some ongoing effort on transforming the implementation of KMeans to Matrix multiplication.

[AmplabJenkins]

Can one of the admins verify this patch?