Uh oh!
There was an error while loading. Please reload this page.
[SPARK-27464][CORE] Added Constant instead of referring string literal used from many places - #24368
[SPARK-27464][CORE] Added Constant instead of referring string literal used from many places#24368shivusondur wants to merge 3 commits into
Conversation
HyukjinKwon
commented
Apr 14, 2019
ok to test |
| .createOptional | ||
| private[spark] val BUFFER_PAGESIZE = ConfigBuilder("spark.buffer.pageSize") | ||
| .bytesConf(ByteUnit.BYTE) |
There was a problem hiding this comment.
@HyukjinKwon
Thanks for your time.
Now I added the doc and corrected the scalastyle issue
There was a problem hiding this comment.
I was going to say this needs a default, but it looks like the default in non-test code is not a single value, and different tests want different defaults.
SparkQA
commented
Apr 14, 2019
Test build #104570 has finished for PR 24368 at commit
|
> corrected the scalasyle issue
| .createOptional | ||
| private[spark] val BUFFER_PAGESIZE = ConfigBuilder("spark.buffer.pageSize") | ||
| .bytesConf(ByteUnit.BYTE) |
There was a problem hiding this comment.
I was going to say this needs a default, but it looks like the default in non-test code is not a single value, and different tests want different defaults.
| val size = ByteArrayMethods.nextPowerOf2(maxTungstenMemory / cores / safetyFactor) | ||
| val default = math.min(maxPageSize, math.max(minPageSize, size)) | ||
| conf.getSizeAsBytes("spark.buffer.pageSize", default) | ||
| conf.getSizeAsBytes(BUFFER_PAGESIZE.key, default) |
There was a problem hiding this comment.
If this is a byteConf, can this just use conf.get()?
There was a problem hiding this comment.
@srowen
Updated the code to use conf.get.
initially, I thought the same, But there is no method to provide the default value sparkConf class.
Now I am checking the values, if not present initializing with a default value.
SparkQA
commented
Apr 14, 2019
Test build #104571 has finished for PR 24368 at commit
|
SparkQA
commented
Apr 14, 2019
Test build #104577 has finished for PR 24368 at commit
|
attilapiros
left a comment
There was a problem hiding this comment.
Small question otherwise LGTM.
Uh oh!
There was an error while loading. Please reload this page.
commented
Apr 16, 2019
Merged to master |
commented
Apr 16, 2019
@srowen |
What changes were proposed in this pull request?
Added Constant instead of referring the same String literal "spark.buffer.pageSize" from many places
How was this patch tested?
Run the corresponding Unit Test Cases manually.