[SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI by shahidki31 · Pull Request #26378 · apache/spark

shahidki31 · 2019-11-03T09:44:02Z

What changes were proposed in this pull request?

Support JDBC/ODBC tab for HistoryServer WebUI. Currently from Historyserver we can't access the JDBC/ODBC tab for thrift server applications. In this PR, I am doing 2 main changes

Refactor existing thrift server listener to support kvstore
Add history server plugin for thrift server listener and tab.

Why are the changes needed?

Users can access Thriftserver tab from History server for both running and finished applications,

Does this PR introduce any user-facing change?

Support for JDBC/ODBC tab for the WEBUI from History server

How was this patch tested?

Add UT and Manual tests

Start Thriftserver and Historyserver

sbin/stop-thriftserver.sh
sbin/stop-historyserver.sh
sbin/start-thriftserver.sh
sbin/start-historyserver.sh

Launch beeline
bin/beeline -u jdbc:hive2://localhost:10000
Run queries

Go to the JDBC/ODBC page of the WebUI from History server

SparkQA · 2019-11-03T09:46:18Z

Test build #113164 has finished for PR 26378 at commit f4a16a5.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2019-11-03T09:54:44Z

Test build #113165 has finished for PR 26378 at commit bff90de.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2019-11-03T10:59:48Z

Test build #113170 has finished for PR 26378 at commit 5490c8e.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-11-03T11:44:35Z

Test build #113166 has finished for PR 26378 at commit dfd1ff6.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2019-11-03T11:47:54Z

Test build #113171 has finished for PR 26378 at commit 24dde37.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class HiveThriftServer2HistoryServerPlugin extends AppHistoryServerPlugin
class HiveThriftServer2AppStatusStore(

shahidki31 · 2019-11-03T11:49:56Z

retest this please

SparkQA · 2019-11-03T12:02:39Z

Test build #113172 has finished for PR 26378 at commit 905e635.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class HiveThriftServer2HistoryServerPlugin extends AppHistoryServerPlugin
class HiveThriftServer2AppStatusStore(

SparkQA · 2019-11-03T14:02:07Z

Test build #113173 has finished for PR 26378 at commit 4eab137.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class HiveThriftServer2HistoryServerPlugin extends AppHistoryServerPlugin
class HiveThriftServer2AppStatusStore(

shahidki31 · 2019-11-04T09:44:38Z

retest this please

SparkQA · 2019-11-04T11:48:48Z

Test build #113204 has finished for PR 26378 at commit 4eab137.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class HiveThriftServer2HistoryServerPlugin extends AppHistoryServerPlugin
class HiveThriftServer2AppStatusStore(

shahidki31 · 2019-11-04T14:59:25Z

Jenkins, retest this please

shahidki31 · 2019-11-04T16:50:28Z

Retest this please

shahidki31

cc @vanzin , @cloud-fan , @wangym , @srowen , @AngersZhuuuu
Kindly review

shahidki31 · 2019-11-04T17:11:56Z

 }

- private[thriftserver] class SessionInfo(
- val sessionId: String,


The code here and below are moved to HiveThriftServer2Listener.scala class

shahidki31 · 2019-11-04T17:12:35Z

- * An inner sparkListener called in sc.stop to clean up the HiveThriftServer2
- */
- private[thriftserver] class HiveThriftServer2Listener(
- val server: HiveServer2,


Moved to a separate file (HiveThriftserver2Listener.scala)

shahidki31 · 2019-11-04T17:14:13Z

+ ip: String,
+ sessionId: String,
+ userName: String,
+ startTime: Long) extends SparkListenerEvent


New events added for Thriftserver applications. Eventually these events will be catch by the listeners EventLoggingListner as well as HiveThriftServer2Listener`

shahidki31 · 2019-11-04T17:16:23Z

 cleanup(OperationState.CLOSED)
- HiveThriftServer2.listener.onOperationClosed(statementId)
+ HiveThriftServer2.listener.postLiveListenerBus(SparkListenerOperationClosed(statementId,
+ System.currentTimeMillis()))


Changes here are below files are,
posting the Thriftserver specific events to LiveListener Bus, so that it can get by both EvenLoggingListener (For HistoryServer) as well as HiveThriftServer2Listener (For live UI)

shahidki31 · 2019-11-04T17:17:41Z

+class HiveThriftServer2AppStatusStore(
+ store: KVStore,
+ val listener: Option[HiveThriftServer2Listener] = None) {
+


Both Live and History UI will read from this store for populating the WEBUI

SparkQA · 2019-11-04T18:37:14Z

Test build #113216 has finished for PR 26378 at commit 4eab137.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class HiveThriftServer2HistoryServerPlugin extends AppHistoryServerPlugin
class HiveThriftServer2AppStatusStore(

SparkQA · 2019-11-04T18:59:48Z

Test build #113217 has finished for PR 26378 at commit 540faf0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class HiveThriftServer2HistoryServerPlugin extends AppHistoryServerPlugin
class HiveThriftServer2AppStatusStore(

shahidki31 · 2019-11-04T19:49:36Z

retest this please

SparkQA · 2019-11-04T21:29:25Z

Test build #113224 has finished for PR 26378 at commit e965c5f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class HiveThriftServer2HistoryServerPlugin extends AppHistoryServerPlugin
class HiveThriftServer2AppStatusStore(

shahidki31 · 2019-11-04T21:34:56Z

Not sure which test is failing. seems all UTs are passing as per the Jenkins report.

SparkQA · 2019-11-04T21:44:29Z

Test build #113225 has finished for PR 26378 at commit e965c5f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class HiveThriftServer2HistoryServerPlugin extends AppHistoryServerPlugin
class HiveThriftServer2AppStatusStore(

SparkQA · 2019-11-05T21:25:21Z

Test build #113284 has finished for PR 26378 at commit 5c2932e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-11-27T01:44:55Z

Test build #114497 has finished for PR 26378 at commit db4269c.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-11-27T03:20:01Z

Test build #114492 has finished for PR 26378 at commit a55927e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-11-27T03:46:14Z

Test build #114495 has finished for PR 26378 at commit af65eed.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

juliuszsompolski

LGTM pending scalastyle.

SparkQA · 2019-11-27T16:48:58Z

Test build #114538 has finished for PR 26378 at commit f7fcc75.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang

LGTM, thanks for the work!

gengliangwang · 2019-11-28T00:08:51Z

@shahidki31 one more comment, I have a good idea about fixing the UI tab order.
We can add a new API displayOrder in the trait AppHistoryServerPlugin(with default value as Int.MaxValue).
And then we can set the displayOrder of SQL tab as 0, and the thriftserver tab as 1

I will merge this one once this is updated.

shahidki31 · 2019-11-28T01:54:06Z

@shahidki31 one more comment, I have a good idea about fixing the UI tab order.
We can add a new API displayOrder in the trait AppHistoryServerPlugin(with default value as Int.MaxValue).
And then we can set the displayOrder of SQL tab as 0, and the thriftserver tab as 1

Yeah, that is a nice idea. Thanks @gengliangwang . I will update the code

…nto ThriftKVStore

SparkQA · 2019-11-28T05:12:30Z

Test build #114558 has finished for PR 26378 at commit 2bc86c5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-11-28T12:05:42Z

Test build #114569 has finished for PR 26378 at commit 39ddc89.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

shahidki31 · 2019-11-29T06:16:32Z

retest this please

SparkQA · 2019-11-29T08:05:02Z

Test build #114603 has finished for PR 26378 at commit 39ddc89.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

shahidki31 · 2019-11-29T08:37:13Z

retest this please

SparkQA · 2019-11-29T11:18:31Z

Test build #114625 has finished for PR 26378 at commit 39ddc89.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang · 2019-11-30T03:43:18Z

Thanks, merging to master

shahidki31 · 2019-11-30T05:05:22Z

Thanks a lot @gengliangwang @juliuszsompolski @AngersZhuuuu for the reviews

…oryServer WebUI ### What changes were proposed in this pull request? Support JDBC/ODBC tab for HistoryServer WebUI. Currently from Historyserver we can't access the JDBC/ODBC tab for thrift server applications. In this PR, I am doing 2 main changes 1. Refactor existing thrift server listener to support kvstore 2. Add history server plugin for thrift server listener and tab. ### Why are the changes needed? Users can access Thriftserver tab from History server for both running and finished applications, ### Does this PR introduce any user-facing change? Support for JDBC/ODBC tab for the WEBUI from History server ### How was this patch tested? Add UT and Manual tests 1. Start Thriftserver and Historyserver ``` sbin/stop-thriftserver.sh sbin/stop-historyserver.sh sbin/start-thriftserver.sh sbin/start-historyserver.sh ``` 2. Launch beeline `bin/beeline -u jdbc:hive2://localhost:10000` 3. Run queries Go to the JDBC/ODBC page of the WebUI from History server ![image](https://user-images.githubusercontent.com/23054875/68365501-cf013700-0156-11ea-84b4-fda8008c92c4.png) Closesapache#26378 from shahidki31/ThriftKVStore. Authored-by: shahid <shahidki31@gmail.com> Signed-off-by: Gengliang Wang <gengliang.wang@databricks.com>

shahidki31 force-pushed the ThriftKVStore branch from dfd1ff6 to 7015ac6Compare November 3, 2019 10:49

shahidki31 force-pushed the ThriftKVStore branch from 5490c8e to 24dde37Compare November 3, 2019 11:46

shahidki31 force-pushed the ThriftKVStore branch from 24dde37 to 905e635Compare November 3, 2019 11:49

shahidki31 mentioned this pull request Nov 3, 2019
[SPARK-29726][WEBUI]Support KVStore for HiveThriftServer2Listener #26372
Closed

shahidki31 force-pushed the ThriftKVStore branch from 905e635 to 4eab137Compare November 3, 2019 12:13

dongjoon-hyun added the WEB UI label Nov 3, 2019

shahidki31 force-pushed the ThriftKVStore branch 2 times, most recently from 841f8a5 to 540faf0Compare November 4, 2019 17:10

shahidki31 commented Nov 4, 2019
View reviewed changes

kvstore thrift
e965c5f

shahidki31 force-pushed the ThriftKVStore branch from 540faf0 to e965c5fCompare November 4, 2019 19:48

gengliangwang reviewed Nov 27, 2019
View reviewed changes

Comment thread...ver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/HiveThriftServer2Listener.scala
Comment thread...src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/HiveThriftServer2EventManager.scala Outdated

address comment
db4269c

juliuszsompolski approved these changes Nov 27, 2019
View reviewed changes

Comment thread...tserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPageSuite.scala Outdated

scalastyle
f7fcc75

gengliangwang approved these changes Nov 27, 2019
View reviewed changes

shahidki31 added 2 commits November 28, 2019 08:19

add display order. Minor refactor test.
5db0667

Merge branch 'ThriftKVStore' ofhttps://github.com/shahidki31/sparki…
2bc86c5
…nto ThriftKVStore

gengliangwang reviewed Nov 28, 2019
View reviewed changes

Comment threadcore/src/main/scala/org/apache/spark/status/AppHistoryServerPlugin.scala Outdated

gengliangwang reviewed Nov 28, 2019
View reviewed changes

Comment threadcore/src/main/scala/org/apache/spark/status/AppHistoryServerPlugin.scala Outdated

gengliangwang reviewed Nov 28, 2019
View reviewed changes

Comment threadsql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLHistoryServerPlugin.scala Outdated

address comments
39ddc89

gengliangwang closed this in b182ed8Nov 30, 2019

shahidki31 deleted the ThriftKVStore branch November 30, 2019 05:50

Uh oh!

Conversation

shahidki31 commented Nov 3, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Nov 3, 2019

Uh oh!

SparkQA commented Nov 3, 2019

Uh oh!

SparkQA commented Nov 3, 2019

Uh oh!

SparkQA commented Nov 3, 2019

Uh oh!

SparkQA commented Nov 3, 2019

Uh oh!

shahidki31 commented Nov 3, 2019

Uh oh!

SparkQA commented Nov 3, 2019

Uh oh!

SparkQA commented Nov 3, 2019

Uh oh!

shahidki31 commented Nov 4, 2019

Uh oh!

SparkQA commented Nov 4, 2019

Uh oh!

shahidki31 commented Nov 4, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shahidki31 commented Nov 4, 2019

Uh oh!

shahidki31 left a comment

Choose a reason for hiding this comment

Uh oh!

shahidki31Nov 4, 2019

Choose a reason for hiding this comment

Uh oh!

shahidki31Nov 4, 2019

Choose a reason for hiding this comment

Uh oh!

shahidki31Nov 4, 2019

Choose a reason for hiding this comment

Uh oh!

shahidki31Nov 4, 2019

Choose a reason for hiding this comment

Uh oh!

shahidki31Nov 4, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 4, 2019

Uh oh!

SparkQA commented Nov 4, 2019

Uh oh!

shahidki31 commented Nov 4, 2019

Uh oh!

SparkQA commented Nov 4, 2019

Uh oh!

shahidki31 commented Nov 4, 2019

Uh oh!

SparkQA commented Nov 4, 2019

Uh oh!

SparkQA commented Nov 5, 2019

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Nov 27, 2019

Uh oh!

SparkQA commented Nov 27, 2019

Uh oh!

SparkQA commented Nov 27, 2019

Uh oh!

juliuszsompolski left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SparkQA commented Nov 27, 2019

shahidki31 commented Nov 3, 2019•
edited
Loading

shahidki31 commented Nov 4, 2019•
edited
Loading

shahidki31Nov 4, 2019•
edited
Loading