Uh oh!
There was an error while loading. Please reload this page.
[SPARK-29530][SQL][2.4] Make SQLConf in SQL parse process thread safe - #26240
[SPARK-29530][SQL][2.4] Make SQLConf in SQL parse process thread safe#26240AngersZhuuuu wants to merge 3 commits into
Conversation
AngersZhuuuu
commented
Oct 24, 2019
@gatorsmile Open pr for branch-2.4. |
HyukjinKwon
commented
Oct 24, 2019
ok to test |
uncleGen
commented
Oct 24, 2019
Could you please add some unit test? |
This pr is to fix problem comment in #26187 (comment) |
SparkQA
commented
Oct 24, 2019
Test build #112591 has finished for PR 26240 at commit
|
AngersZhuuuu
commented
Oct 24, 2019
@gatorsmile@HyukjinKwon@uncleGen |
SparkQA
commented
Oct 24, 2019
Test build #112601 has finished for PR 26240 at commit
|
Uh oh!
There was an error while loading. Please reload this page.
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM. Thank you, @AngersZhuuuu and @HyukjinKwon .
Merged to branch-2.4
### What changes were proposed in this pull request? SparkSession.sql() method parse process not under current sparksession's conf, so some configuration about parser is not valid in multi-thread situation. In this pr, we add a SQLConf parameter to AbstractSqlParser and initial it with SessionState's conf. Then for each SparkSession's parser process. It will use's it's own SessionState's SQLConf and to be thread safe ### Why are the changes needed? Fix bug ### Does this PR introduce any user-facing change? NO ### How was this patch tested? NO Closes#26240 from AngersZhuuuu/SPARK-29530-V2.4. Authored-by: angerszhu <angers.zhu@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
What changes were proposed in this pull request?
SparkSession.sql() method parse process not under current sparksession's conf, so some configuration about parser is not valid in multi-thread situation.
In this pr, we add a SQLConf parameter to AbstractSqlParser and initial it with SessionState's conf.
Then for each SparkSession's parser process. It will use's it's own SessionState's SQLConf and to be thread safe
Why are the changes needed?
Fix bug
Does this PR introduce any user-facing change?
NO
How was this patch tested?
NO