Skip to content

[SPARK-21993][SQL] Close sessionState when finish - #19219

Closed
jinxing64 wants to merge 1 commit into
apache:masterfrom
jinxing64:SPARK-21993
Closed

[SPARK-21993][SQL] Close sessionState when finish#19219
jinxing64 wants to merge 1 commit into
apache:masterfrom
jinxing64:SPARK-21993

Conversation

@jinxing64

@jinxing64jinxing64 commented Sep 13, 2017

Copy link
Copy Markdown

What changes were proposed in this pull request?

In current code, SessionState in SparkSQLCLIDriver is not guaranteed to be closed. As a result, some tmp files/dirs are not removed when job finished. We can close SessionState in shutdown hook.

srowen
srowen reviewed Sep 13, 2017
View reviewed changes
Comment thread...e-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala Outdated

@srowensrowenSep 13, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if it matters, but keep in mind this will hold a reference to sessionState then until the JVM exits

@SparkQA

SparkQA commented Sep 13, 2017

Copy link
Copy Markdown

Test build #81713 has finished for PR 19219 at commit 2a06c0c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jinxing64

jinxing64 commented Sep 14, 2017•
edited
Loading

Copy link
Copy Markdown
Author

@gatorsmile@HyukjinKwon
Could you please give some advice on this?

@jinxing64

jinxing64 commented Sep 14, 2017

Copy link
Copy Markdown
Author

It seems there are still some other places where session state is not guaranteed to be closed. I will update this pr soon.

@jinxing64jinxing64 changed the title [SPARK-21993][SQL] Close sessionState in shutdown hook.[SPARK-21993][SQL][WIP] Close sessionState when finishSep 16, 2017
@SparkQA

SparkQA commented Sep 16, 2017

Copy link
Copy Markdown

Test build #81838 has finished for PR 19219 at commit 1187dac.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

cenyuhai
cenyuhai reviewed Sep 16, 2017
View reviewed changes
Comment threadsql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionStateBuilder.scala Outdated

@cenyuhaicenyuhaiSep 16, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newSession is to isolate SparkSession in Spark ThriftServer, You must judge whether is thriftserver or not.

@SparkQA

SparkQA commented Sep 18, 2017

Copy link
Copy Markdown

Test build #81874 has finished for PR 19219 at commit d7a484d.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Sep 20, 2017

Copy link
Copy Markdown

Test build #81997 has finished for PR 19219 at commit df49a19.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Sep 21, 2017

Copy link
Copy Markdown

Test build #82017 has finished for PR 19219 at commit e421113.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jinxing64jinxing64 changed the title [SPARK-21993][SQL][WIP] Close sessionState when finish[SPARK-21993][SQL] Close sessionState when finishSep 21, 2017
@gatorsmile

gatorsmile commented Feb 1, 2018

Copy link
Copy Markdown
Member

cc @liufengdb

@liufengdb

liufengdb commented Feb 1, 2018

Copy link
Copy Markdown

The major issue this PR tries to cover has been fixed by #20029, so I think we are good if there are no calls to HiveClientImpl.newSession. We can close this PR with no-fix.

@SparkQA

SparkQA commented Feb 1, 2018

Copy link
Copy Markdown

Test build #86911 has finished for PR 19219 at commit e421113.

  • This patch fails PySpark unit tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@srowensrowen mentioned this pull request Oct 24, 2018
Closed
zifeif2 pushed a commit to zifeif2/spark that referenced this pull request Nov 22, 2025
@srowen@zifeif2
[BUILD] Close stale PRs
25b8667
Closesapache#22567Closesapache#18457Closesapache#21517Closesapache#21858Closesapache#22383Closesapache#19219Closesapache#22401Closesapache#22811Closesapache#20405Closesapache#21933Closesapache#22819 from srowen/ClosePRs.
Authored-by: Sean Owen <sean.owen@databricks.com>
Signed-off-by: Sean Owen <sean.owen@databricks.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@srowensrowensrowen left review comments
+1 more reviewer
@cenyuhaicenyuhaicenyuhai left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@jinxing64@SparkQA@gatorsmile@liufengdb@srowen@cenyuhai