Uh oh!
There was an error while loading. Please reload this page.
[SPARK-1097] Workaround Hadoop conf ConcurrentModification issue - #1273
[SPARK-1097] Workaround Hadoop conf ConcurrentModification issue#1273colorant wants to merge 2 commits into
Conversation
AmplabJenkins
commented
Jul 1, 2014
Merged build triggered. |
AmplabJenkins
commented
Jul 1, 2014
Merged build started. |
colorant
commented
Jul 1, 2014
as described in #1000 |
AmplabJenkins
commented
Jul 1, 2014
Merged build finished. All automated tests passed. |
AmplabJenkins
commented
Jul 1, 2014
All automated tests passed. |
There was a problem hiding this comment.
Can we just use "conf" here and on the next line, instead of broadcastedConf.value.value? This is actually the first guy that assumes conf is not null, though, maybe add an assert for that.
There was a problem hiding this comment.
@aarondav , Yes, I also thought about this before. The reason I keep use broadcastedConf.value.value here is because: though broadcast variable is suggested to be read only and not changed, But I wonder maybe in case someone miss use it and change the value, read the latest value might be helpful. And it read the latest code in the next line in the original code, so I keep this style. But think again, if the value did changed in some place without hold any synchronize lock, this might still not be able to solve the problem. I will update the pull request.
There was a problem hiding this comment.
regard conf , it is wrapped in SerializableWritable which enforce it to be a Writable, I think it won't be a null , or exception already been thrown somewhere before here. What do you think?
There was a problem hiding this comment.
We can keep without the check, the NPE should be apparent on the synchronized block.
AmplabJenkins
commented
Jul 4, 2014
Merged build triggered. |
AmplabJenkins
commented
Jul 4, 2014
Merged build started. |
AmplabJenkins
commented
Jul 4, 2014
Merged build finished. All automated tests passed. |
AmplabJenkins
commented
Jul 4, 2014
All automated tests passed. |
aarondav
commented
Jul 4, 2014
LGTM, merging into master and branch-1.0. |
Workaround Hadoop conf ConcurrentModification issue Author: Raymond Liu <raymond.liu@intel.com> Closes#1273 from colorant/hadoopRDD and squashes the following commits: 994e98b [Raymond Liu] Address comments e2cda3d [Raymond Liu] Workaround Hadoop conf ConcurrentModification issue (cherry picked from commit 5fa0a05) Signed-off-by: Aaron Davidson <aaron@databricks.com>
Workaround Hadoop conf ConcurrentModification issue Author: Raymond Liu <raymond.liu@intel.com> Closesapache#1273 from colorant/hadoopRDD and squashes the following commits: 994e98b [Raymond Liu] Address comments e2cda3d [Raymond Liu] Workaround Hadoop conf ConcurrentModification issue
Workaround Hadoop conf ConcurrentModification issue