Uh oh!
There was an error while loading. Please reload this page.
[SPARK-18471][MLLIB][BACKPORT-2.0] In LBFGS, avoid sending huge vectors of 0 - #16279
[SPARK-18471][MLLIB][BACKPORT-2.0] In LBFGS, avoid sending huge vectors of 0 #16279AnthonyTruchet wants to merge 1 commit into
Conversation
…rs of 0 CostFun used to send a dense vector of zeroes as a closure in a treeAggregate call. To avoid that, we replace treeAggregate by mapPartition + treeReduce, creating a zero vector inside the mapPartition block in-place. Unit test for module mllib run locally for correctness. As for performance we run an heavy optimization on our production data (50 iterations on 128 MB weight vectors) and have seen significant decrease in terms both of runtime and container being killed by lack of off-heap memory. Author: Anthony Truchet <a.truchet@criteo.com> Author: sethah <seth.hendrickson16@gmail.com>
srowen
commented
Dec 14, 2016
I don't think we would backport this to even 2.1. You can close this. |
AnthonyTruchet
commented
Dec 14, 2016
May I ask why ? There was no conflicts so no additional qualification work is required and this looks like a performance bug fix to me, not a new feature. In order to adjust our contribution policy (fix to our internal version vs pushing upstream) we would need to understand better the backporting policy: would you have any more detailed pointer at it ? |
srowen
commented
Dec 14, 2016
The guidance such as it is is here: http://spark.apache.org/versioning-policy.html |
AnthonyTruchet
commented
Dec 15, 2016
Ok, thanks for the pointer. I do agree this is a non critical judgement call . |
sethah
commented
Jan 10, 2017
Can we close it? |
AnthonyTruchet
commented
Jan 10, 2017
We have backported it to our internal version of Spark anyhow. As told above feel free to close it if you consider this is not worth officially backporting :-) |
SparkQA
commented
Jan 14, 2017
Test build #3534 has finished for PR 16279 at commit
|
Closesapache#15736Closesapache#16309Closesapache#16485Closesapache#16502Closesapache#16196Closesapache#16498Closesapache#12380Closesapache#16764Closesapache#14394Closesapache#14204Closesapache#14027Closesapache#13690Closesapache#16279 Author: Sean Owen <sowen@cloudera.com> Closesapache#16778 from srowen/CloseStalePRs.
Backport #16037 to 2.0 branch
What changes were proposed in this pull request?
CostFun used to send a dense vector of zeroes as a closure in a
treeAggregate call. To avoid that, we change the aggregation operations
to convert sparse vectors into dense vectors on the fly if needed and we
pass a sparse 0 vector which is lightweight.
How was this patch tested?
Unit test for module mllib run locally for correctness.
As for performance we run an heavy optimization on our production data (50 iterations on 128 MB weight vectors) and have seen significant decrease in terms both of runtime and container being killed by lack of off-heap memory.
Author: Anthony Truchet a.truchet@criteo.com
Author: sethah seth.hendrickson16@gmail.com