Uh oh!
There was an error while loading. Please reload this page.
[SPARK-37084][SQL] Set spark.sql.files.openCostInBytes to bytesConf - #34353
[SPARK-37084][SQL] Set spark.sql.files.openCostInBytes to bytesConf#34353RabbidHY wants to merge 2 commits into
Conversation
| .longConf | ||
| .createWithDefault(4 * 1024 * 1024) | ||
| .bytesConf(ByteUnit.BYTE) | ||
| .createWithDefaultString("4MB") |
There was a problem hiding this comment.
Can we add a simple test at ConfigEntrySuite to make sure byte configuration is able to take the long type numbers?
There was a problem hiding this comment.
+1; we could probably extend this existing test:
HyukjinKwon
commented
Oct 21, 2021
Let's also file a JIRA, see also https://spark.apache.org/contributing.html |
HyukjinKwon
commented
Oct 23, 2021
ok to test |
HyukjinKwon
commented
Oct 23, 2021
Merged to master. |
SparkQA
commented
Oct 23, 2021
Kubernetes integration test starting |
SparkQA
commented
Oct 23, 2021
Kubernetes integration test status failure |
SparkQA
commented
Oct 23, 2021
Test build #144550 has finished for PR 34353 at commit
|
AmplabJenkins
commented
Oct 23, 2021
Can one of the admins verify this patch? |
### What changes were proposed in this pull request? Set `spark.sql.files.openCostInBytes` to bytesConf. ### Why are the changes needed? The name is _*InBytes_, but it actually only accepts **long type**. This is confusing for users. After the changes, it can accept **String** as input which is more flexible to users. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing test. Closes#34353 from RabbidHY/SPARK-37084. Authored-by: RabbidHY <stitch106hy@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
What changes were proposed in this pull request?
Set
spark.sql.files.openCostInBytesto bytesConf.Why are the changes needed?
The name is *InBytes, but it actually only accepts long type. This is confusing for users. After the changes, it can accept String as input which is more flexible to users.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Existing test.