Uh oh!
There was an error while loading. Please reload this page.
[SPARK-20443][MLLIB][ML] set ALS blockify size - #17739
Conversation
SparkQA
commented
Apr 24, 2017
Test build #76096 has started for PR 17739 at commit |
MLnick
commented
Apr 24, 2017
Just to confirm, the #users is 48 million, #items is 1.7 million? |
MLnick
commented
Apr 24, 2017
Or is it 48,000 and 1,700? |
users is 480,000, items is 17,000. Thanks |
MLnick
commented
Apr 24, 2017
ok. And it is the timing for |
mpjlu
commented
Apr 24, 2017
RecommandProductsForUsers. Thanks |
MLnick
commented
Apr 24, 2017
It's interesting to see the performance difference. I've also been looking at performance of recommend all but haven't gotten to varying the block sizes just yet. I'm potentially in favor of exposing it as a param - but what you've got here doesn't do anything to the public API so how does that help? |
mpjlu
commented
Apr 24, 2017
SparkQA
commented
Apr 24, 2017
Test build #76108 has finished for PR 17739 at commit
|
SparkQA
commented
May 11, 2017
Test build #76775 has finished for PR 17739 at commit
|
SparkQA
commented
May 11, 2017
Test build #76781 has started for PR 17739 at commit |
SparkQA
commented
May 11, 2017
Test build #76787 has finished for PR 17739 at commit
|
SparkQA
commented
May 16, 2017
Test build #76971 has finished for PR 17739 at commit
|
mpjlu
commented
May 17, 2017
retest this please |
SparkQA
commented
May 17, 2017
Test build #76998 has finished for PR 17739 at commit
|
mpjlu
commented
Jan 15, 2018
Because I don't have the environment to continue this work, I will close it. Thanks. |
What changes were proposed in this pull request?
The blockSize of MLLIB ALS is very important for ALS performance.
In our test, when the blockSize is 128, the performance is about 4X comparing with the blockSize is 4096 (default value).
The following are our test results:
BlockSize(recommendationForAll time)
128(124s), 256(160s), 512(184s), 1024(244s), 2048(332s), 4096(488s), 8192(OOM)
The Test Environment:
3 workers: each work 10 core, each work 30G memory, each work 1 executor.
The Data: User 480,000, and Item 17,000
How was this patch tested?
The existing UT