[SPARK-5100][SQL]add webui for thriftserver by tianyi · Pull Request #5730 · apache/spark

tianyi · 2015-04-28T01:40:18Z

This PR is a rebased version of #3946 , and mainly focused on creating an independent tab for the thrift server in spark web UI.

Features:

Session related statistics ( username and IP are only supported in hive-0.13.1 )
List all the SQL executing or executed on this server
Provide links to the job generated by SQL
Provide link to show all SQL executing or executed in a specified session

Prototype snapshots:

This is the main page for thrift server

SparkQA · 2015-04-28T01:43:45Z

Test build #31097 has started for PR 5730 at commit 9830199.

liancheng · 2015-04-28T03:01:51Z

totalExecution?

SparkQA · 2015-04-28T03:22:42Z

Test build #31097 has finished for PR 5730 at commit 9830199.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class HiveThriftServer2Listener(
This patch does not change any dependencies.

liancheng · 2015-04-28T03:24:40Z

Append () to the method name since it causes side effect.

Same as above.

WangTaoTheTonic · 2015-04-28T03:58:09Z

Sorry for not following the previous comments but could you tell the difference between this tab and Thrift Server's driver UI?
Once this is added, will Thrift Server's Driver UI still be reserved?

liancheng · 2015-04-28T04:04:59Z

Maybe onStatementCompiled? Also, executePlan should be executionPlan.

tianyi · 2015-04-28T04:59:23Z

@WangTaoTheTonic , the Thrift Server's driver UI you mentioned is a universal UI for all kinds of work on spark, and mainly focus on jobs, stages and tasks.
In this PR, we provide a new tab that focus on users, sessions and SQL.
So the driver UI won't be replaced since they are entirely different.

liancheng · 2015-04-28T05:11:38Z

Import order is wrong here.

liancheng · 2015-04-28T05:19:08Z

This generally looks good, left some comments, most are about styling issues.

@JoshRosen Would you mind to double check the web UI changes? This should be generally the same as #3946. Thanks! @tianyi will add Selenium based test cases later.

SparkQA · 2015-04-28T07:17:46Z

Test build #31128 has started for PR 5730 at commit c9df6f9.

tianyi · 2015-04-28T07:21:21Z

@JoshRosen I just add some tests for thrift server UI. But I'm not sure if I need to add test cases for all the feature. Any suggestion?

SparkQA · 2015-04-28T07:23:44Z

Test build #31129 has started for PR 5730 at commit 8809157.

SparkQA · 2015-04-28T07:39:44Z

Test build #31128 has finished for PR 5730 at commit c9df6f9.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-28T07:48:10Z

Test build #31129 has finished for PR 5730 at commit 8809157.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-28T08:08:54Z

Test build #31132 has started for PR 5730 at commit aa20408.

SparkQA · 2015-04-28T09:30:56Z

Test build #31132 has finished for PR 5730 at commit aa20408.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-29T06:17:13Z

Merged build triggered.

AmplabJenkins · 2015-04-29T06:17:21Z

Merged build started.

SparkQA · 2015-04-29T06:17:58Z

Test build #31254 has started for PR 5730 at commit 0efe3d5.

SparkQA · 2015-04-29T06:42:20Z

Test build #31254 has finished for PR 5730 at commit 0efe3d5.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-29T06:42:21Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-04-29T06:42:22Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31254/
Test FAILed.

AmplabJenkins · 2015-04-29T07:02:09Z

Merged build triggered.

AmplabJenkins · 2015-04-29T07:02:16Z

Merged build started.

SparkQA · 2015-04-29T07:02:48Z

Test build #31256 has started for PR 5730 at commit cfd14c7.

SparkQA · 2015-04-29T09:42:22Z

Test build #31256 has finished for PR 5730 at commit cfd14c7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-29T09:42:27Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-04-29T09:42:28Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31256/
Test PASSed.

tianyi · 2015-04-29T09:52:27Z

@liancheng The test codes and is ready to review.

liancheng · 2015-05-04T08:56:43Z

LGTM, merging to master. Thanks for working on this!

marmbrus · 2015-05-06T18:32:14Z

The test added here is really noisy:

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/04/29 01:59:46 INFO UISeleniumSuite: Trying to start HiveThriftServer2: port=19866, mode=binary, attempt=0
15/04/29 02:00:16 INFO UISeleniumSuite: HiveThriftServer2 started successfully
15/04/29 02:00:19 WARN DefaultCssErrorHandler: CSS error: 'http://localhost:24602/static/bootstrap.min.css' [10:11] Error in style rule. (Invalid token "*". Was expecting one of: <EOF>, <S>, <IDENT>, "}", ";".)
15/04/29 02:00:19 WARN DefaultCssErrorHandler: CSS warning: 'http://localhost:24602/static/bootstrap.min.css' [10:11] Ignoring the following declarations in this rule.
15/04/29 02:00:19 WARN DefaultCssErrorHandler: CSS error: 'http://localhost:24602/static/bootstrap.min.css' [15:41] Error in style rule. (Invalid token "*". Was expecting one of: <EOF>, <S>, <IDENT>, "}", ";".)
15/04/29 02:00:19 WARN DefaultCssErrorHandler: CSS warning: 'http://localhost:24602/static/bootstrap.min.css' [15:41] Ignoring the following declarations in this rule.
15/04/29 02:00:19 WARN DefaultCssErrorHandler: CSS error: 'http://localhost:24602/static/bootstrap.min.css' [26:14] Error in style rule. (Invalid token "*". Was expecting one of: <EOF>, <S>, <IDENT>, "}", ";".)
15/04/29 02:00:19 WARN DefaultCssErrorHandler: CSS warning: 'http://localhost:24602/static/bootstrap.min.css' [26:14] Ignoring the following declarations in this rule.
15/04/29 02:00:19 WARN DefaultCssErrorHandler: CSS error: 'http://localhost:24602/static/bootstrap.min.css' [39:24] Error in style rule. (Invalid token "*". Was expecting one of: <EOF>, <S>, <IDENT>, "}", ";".)

Can you submit a followup to quiet it down?

tianyi · 2015-05-07T03:39:22Z

OK, I will work on it.

tianyi · 2015-05-08T08:02:41Z

@marmbrus the test issue is fixed in #5983

This PR is a rebased version of apache#3946 , and mainly focused on creating an independent tab for the thrift server in spark web UI. Features: 1. Session related statistics ( username and IP are only supported in hive-0.13.1 ) 2. List all the SQL executing or executed on this server 3. Provide links to the job generated by SQL 4. Provide link to show all SQL executing or executed in a specified session Prototype snapshots: This is the main page for thrift server ![image](https://cloud.githubusercontent.com/assets/1411869/7361379/df7dcc64-ed89-11e4-9964-4df0b32f475e.png) Author: tianyi <tianyi.asiainfo@gmail.com> Closesapache#5730 from tianyi/SPARK-5100 and squashes the following commits: cfd14c7 [tianyi] style fix 0efe3d5 [tianyi] revert part of pom change c0f2fa0 [tianyi] extends HiveThriftJdbcTest to start/stop thriftserver for UI test aa20408 [tianyi] fix style problem c9df6f9 [tianyi] add testsuite for thriftserver ui and fix some style issue 9830199 [tianyi] add webui for thriftserver

add webui for thriftserver
9830199

tianyi changed the title ~~add webui for thriftserver~~[SPARK-5100][SQL]add webui for thriftserverApr 28, 2015

tianyi mentioned this pull request Apr 28, 2015
[SPARK-5100][SQL] add thriftserver-ui support #3946
Closed

liancheng reviewed Apr 28, 2015
View reviewed changes

add testsuite for thriftserver ui and fix some style issue
c9df6f9

fix style problem
aa20408

tianyi force-pushed the SPARK-5100 branch from 8809157 to aa20408Compare April 28, 2015 08:04

tianyi added 2 commits April 29, 2015 14:12

extends HiveThriftJdbcTest to start/stop thriftserver for UI test
c0f2fa0

revert part of pom change
0efe3d5

style fix
cfd14c7

asfgit closed this in 343d3bfMay 4, 2015

Uh oh!

Conversation

tianyi commented Apr 28, 2015

Uh oh!

SparkQA commented Apr 28, 2015

Uh oh!

lianchengApr 28, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 28, 2015

Uh oh!

lianchengApr 28, 2015

Choose a reason for hiding this comment

Uh oh!

lianchengApr 28, 2015

Choose a reason for hiding this comment

Uh oh!

WangTaoTheTonic commented Apr 28, 2015

Uh oh!

lianchengApr 28, 2015

Choose a reason for hiding this comment

Uh oh!

tianyi commented Apr 28, 2015

Uh oh!

lianchengApr 28, 2015

Choose a reason for hiding this comment

Uh oh!

liancheng commented Apr 28, 2015

Uh oh!

SparkQA commented Apr 28, 2015

Uh oh!

tianyi commented Apr 28, 2015

Uh oh!

SparkQA commented Apr 28, 2015

Uh oh!

SparkQA commented Apr 28, 2015

Uh oh!

SparkQA commented Apr 28, 2015

Uh oh!

SparkQA commented Apr 28, 2015

Uh oh!

SparkQA commented Apr 28, 2015

Uh oh!

AmplabJenkins commented Apr 29, 2015

Uh oh!

AmplabJenkins commented Apr 29, 2015

Uh oh!

SparkQA commented Apr 29, 2015

Uh oh!

SparkQA commented Apr 29, 2015

Uh oh!

AmplabJenkins commented Apr 29, 2015

Uh oh!

AmplabJenkins commented Apr 29, 2015

Uh oh!

AmplabJenkins commented Apr 29, 2015

Uh oh!

AmplabJenkins commented Apr 29, 2015

Uh oh!

SparkQA commented Apr 29, 2015

Uh oh!

SparkQA commented Apr 29, 2015

Uh oh!

AmplabJenkins commented Apr 29, 2015

Uh oh!

AmplabJenkins commented Apr 29, 2015

Uh oh!

tianyi commented Apr 29, 2015

Uh oh!

liancheng commented May 4, 2015

Uh oh!

marmbrus commented May 6, 2015

Uh oh!

tianyi commented May 7, 2015

Uh oh!

tianyi commented May 8, 2015

Uh oh!

Reviewers

Assignees

Labels