Uh oh!
There was an error while loading. Please reload this page.
[SPARK-35783][SQL] Set the list of read columns in the task configuration to reduce reading of ORC data. - #32923
[SPARK-35783][SQL] Set the list of read columns in the task configuration to reduce reading of ORC data.#32923weixiuli wants to merge 1 commit into
Conversation
…tion to reduce reading of ORC data.
dongjoon-hyun
commented
Jun 17, 2021
ok to test |
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM. Thank you, @weixiuli .
Merged to master for Apache Spark 3.2.0.
weixiuli
commented
Jun 17, 2021
Thank you @dongjoon-hyun for the quick review and comments. |
zhengruifeng
commented
Jun 17, 2021
@weixiuli Great Catch! @dongjoon-hyun maybe we need to backport it to 3.0 & 3.1? |
For me, this is a performance improvement, @zhengruifeng . |
cloud-fan
commented
Jun 23, 2021
Does this mean we never do column pruning for ORC before this PR? And shall we update the result of |
zhengruifeng
commented
Jun 24, 2021
@cloud-fan We are migrating from 2.4.7 to 3.0.2, and observed a significant regression in some cases due to this issue.
|
weixiuli
commented
Jun 24, 2021
@cloud-fan Yes, i will check the |
cloud-fan
commented
Jun 24, 2021
I think this is a serious perf regression we should backport. @dongjoon-hyun what do you think? |
dongjoon-hyun
commented
Jun 24, 2021
Got it. In that case, I'm okay for backporting, @cloud-fan . I'll backport this. |
…tion to reduce reading of ORC data ### What changes were proposed in this pull request? Set the list of read columns in the task configuration to reduce reading of ORC data. ### Why are the changes needed? Now, the ORC reader will read all columns of the ORC table when the task configuration does not set the list of read columns . Therefore, we should set the list of read columns in the task configuration to reduce reading of ORC data. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? exist unittests Closes#32923 from weixiuli/SPARK-35783. Authored-by: weixiuli <weixiuli@jd.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 947c7ea) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
…tion to reduce reading of ORC data ### What changes were proposed in this pull request? Set the list of read columns in the task configuration to reduce reading of ORC data. ### Why are the changes needed? Now, the ORC reader will read all columns of the ORC table when the task configuration does not set the list of read columns . Therefore, we should set the list of read columns in the task configuration to reduce reading of ORC data. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? exist unittests Closes#32923 from weixiuli/SPARK-35783. Authored-by: weixiuli <weixiuli@jd.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 947c7ea) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
…tion to reduce reading of ORC data ### What changes were proposed in this pull request? Set the list of read columns in the task configuration to reduce reading of ORC data. ### Why are the changes needed? Now, the ORC reader will read all columns of the ORC table when the task configuration does not set the list of read columns . Therefore, we should set the list of read columns in the task configuration to reduce reading of ORC data. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? exist unittests Closesapache#32923 from weixiuli/SPARK-35783. Authored-by: weixiuli <weixiuli@jd.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 947c7ea) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
sleep1661
commented
Mar 4, 2022
|
What changes were proposed in this pull request?
Set the list of read columns in the task configuration to reduce reading of ORC data.
Why are the changes needed?
Now, the ORC reader will read all columns of the ORC table when the task configuration does not set the list of read columns . Therefore, we should set the list of read columns in the task configuration to reduce reading of ORC data.
Does this PR introduce any user-facing change?
No
How was this patch tested?
exist unittests