[SPARK-29530][SQL] Make SQLConf in SQL parse process thread safe by AngersZhuuuu · Pull Request #26187 · apache/spark

AngersZhuuuu · 2019-10-21T06:21:51Z

What changes were proposed in this pull request?

As I have comment in SPARK-29516
SparkSession.sql() method parse process not under current sparksession's conf, so some configuration about parser is not valid in multi-thread situation.

In this pr, we add a SQLConf parameter to AbstractSqlParser and initial it with SessionState's conf.
Then for each SparkSession's parser process. It will use's it's own SessionState's SQLConf and to be thread safe

Why are the changes needed?

Fix bug

Does this PR introduce any user-facing change?

NO

How was this patch tested?

NO

AngersZhuuuu · 2019-10-21T06:22:58Z

cc @wangyum @cloud-fan @gatorsmile

wangyum · 2019-10-21T06:31:53Z

ok to test

SparkQA · 2019-10-21T07:05:02Z

Test build #112364 has finished for PR 26187 at commit f456207.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-10-21T07:08:20Z

retest this please

zsxwing · 2019-10-21T07:22:58Z

 * @since 2.0.0
 */
 def sql(sqlText: String): DataFrame = {
+ SparkSession.setActiveSession(this)


Who takes responsibility to set this back if we set the active session in this method? For example, the current active session is spark1 when calling spark2.sql(...), then the active session will become spark2. When we will set it back to spark1, or it will always be spark2?

Who takes responsibility to set this back if we set the active session in this method? For example, the current active session is spark1 when calling spark2.sql(...), then the active session will become spark2. When we will set it back to spark1, or it will always be spark2?
Under normal circumstances， our spark program use one SparkSession, if some one write program use multi sparksession, they should control these things.
And for analyzer, it will set this.
spark/sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
Line 64 in e99a9f7
SparkSession.setActiveSession(sparkSession)
And only
SparkSession.cleanupAnyExistingSession() will call
SparkSession.clearActiveSession() SparkSession.clearDefaultSession()

cloud-fan · 2019-10-21T07:31:09Z

+ SparkSession.setActiveSession(this)
 val tracker = new QueryPlanningTracker
 val plan = tracker.measurePhase(QueryPlanningTracker.PARSING) {
 sessionState.sqlParser.parsePlan(sqlText)


I know that we use SQLConf.get a lot in analyzer/optimizer rules, but does it also true for the parser?

I know that we use SQLConf.get a lot in analyzer/optimizer rules, but does it also true for the parser?
Parser use SQLConf.get too.
It truly make some configuration about parser mess in SparkThriftServer multi thread mode.
spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParseDriver.scala
Line 94 in e99a9f7
lexer.legacy_setops_precedence_enbled =SQLConf.get.setOpsPrecedenceEnforced
spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParseDriver.scala
Line 95 in e99a9f7
lexer.ansi =SQLConf.get.ansiEnabled
It is all about error happened in #26172

shall we define a conf: SQLConf method in ParserDriver so that we can avoid calling SQLConf.get in the parser?

shall we define a conf: SQLConf method in ParserDriver so that we can avoid calling SQLConf.get in the parser?
How about new change? Test is ok for #26172 . also cc @juliuszsompolski

SparkQA · 2019-10-21T10:15:27Z

Test build #112368 has finished for PR 26187 at commit f456207.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-10-21T10:33:29Z

retest this please

SparkQA · 2019-10-21T14:18:38Z

Test build #112383 has finished for PR 26187 at commit f456207.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2019-10-21T16:28:40Z

LGTM, can you update the PR title?

SparkQA · 2019-10-21T18:36:28Z

Test build #112399 has finished for PR 26187 at commit ad9dba8.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class AbstractSqlParser(conf: SQLConf) extends ParserInterface with Logging
class CatalystSqlParser(conf: SQLConf) extends AbstractSqlParser(conf)
class SparkSqlParser(conf: SQLConf) extends AbstractSqlParser(conf)

AngersZhuuuu · 2019-10-22T01:46:08Z

LGTM, can you update the PR title?

PR Title ok now? brief and show purpose.

cloud-fan · 2019-10-22T02:38:26Z

thanks, merging to master!

gatorsmile · 2019-10-24T02:42:52Z

@AngersZhuuuu Could you open a PR against 2.4 branches?

AngersZhuuuu · 2019-10-24T02:55:23Z

@AngersZhuuuu Could you open a PR against 2.4 branches?

I will do this.

Update SparkSession.scala
f456207

wangyum changed the title ~~Reset active SparkSession before parse sql in SparkSession~~[SPARK-29530][SQL] Reset active SparkSession before parse sql in SparkSessionOct 21, 2019

zsxwing reviewed Oct 21, 2019
View reviewed changes

cloud-fan reviewed Oct 21, 2019
View reviewed changes

AngersZhuuuu mentioned this pull request Oct 21, 2019
[SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously #26172
Closed

follow comment
ad9dba8

AngersZhuuuu changed the title ~~[SPARK-29530][SQL] Reset active SparkSession before parse sql in SparkSession~~[SPARK-29530][SQL] SQLParser use SQLConf parameter instead of SQLConf.get in parser processOct 21, 2019

AngersZhuuuu changed the title ~~[SPARK-29530][SQL] SQLParser use SQLConf parameter instead of SQLConf.get in parser process~~[SPARK-29530][SQL] SQLParser use SessionState's SQLConf variable instead of SQLConf.get in parser processOct 21, 2019

AngersZhuuuu changed the title ~~[SPARK-29530][SQL] SQLParser use SessionState's SQLConf variable instead of SQLConf.get in parser process~~[SPARK-29530][SQL] Add SessionState's conf as AbstractSqlParser's parameter instead of SQLConf.get in parser process make it thread safeOct 21, 2019

AngersZhuuuu changed the title ~~[SPARK-29530][SQL] Add SessionState's conf as AbstractSqlParser's parameter instead of SQLConf.get in parser process make it thread safe~~[SPARK-29530][SQL] Make SQLConf in SQL parse process thread safeOct 22, 2019

HyukjinKwon approved these changes Oct 22, 2019
View reviewed changes

cloud-fan closed this in 484f93eOct 22, 2019

AngersZhuuuu mentioned this pull request Oct 24, 2019
[SPARK-29530][SQL][2.4] Make SQLConf in SQL parse process thread safe #26240
Closed

dongjoon-hyun added the SQL label Feb 5, 2020

Uh oh!

Conversation

AngersZhuuuu commented Oct 21, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

AngersZhuuuu commented Oct 21, 2019

Uh oh!

wangyum commented Oct 21, 2019

Uh oh!

SparkQA commented Oct 21, 2019

Uh oh!

wangyum commented Oct 21, 2019

Uh oh!

zsxwingOct 21, 2019

Choose a reason for hiding this comment

Uh oh!

AngersZhuuuuOct 21, 2019

Choose a reason for hiding this comment

Uh oh!

cloud-fanOct 21, 2019

Choose a reason for hiding this comment

Uh oh!

AngersZhuuuuOct 21, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fanOct 21, 2019

Choose a reason for hiding this comment

Uh oh!

AngersZhuuuuOct 21, 2019

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 21, 2019

Uh oh!

HyukjinKwon commented Oct 21, 2019

Uh oh!

SparkQA commented Oct 21, 2019

Uh oh!

cloud-fan commented Oct 21, 2019

Uh oh!

SparkQA commented Oct 21, 2019

Uh oh!

AngersZhuuuu commented Oct 22, 2019

Uh oh!

cloud-fan commented Oct 22, 2019

Uh oh!

gatorsmile commented Oct 24, 2019

Uh oh!

AngersZhuuuu commented Oct 24, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

AngersZhuuuu commented Oct 21, 2019•
edited
Loading

AngersZhuuuuOct 21, 2019•
edited
Loading