[SPARK-8656][WebUI] Fix the webUI and JSON API number is not synced by thegiive · Pull Request #7038 · apache/spark

thegiive · 2015-06-26T10:31:37Z

Spark standalone master web UI show "Alive Workers" total core, total used cores and "Alive workers" total memory, memory used.
But the JSON API page "http://MASTERURL:8088/json" shows "ALL workers" core, memory number.
This webUI data is not sync with the JSON API.
The proper way is to sync the number with webUI and JSON API.

AmplabJenkins · 2015-06-26T10:32:11Z

Can one of the admins verify this patch?

srowen · 2015-06-26T11:02:05Z

This is changing the semantics of the response though. I don't think you can necessarily do this. It is not necessarily true that what is presented in the web UI for human consumption is identical to what's in the JSON response for machine consumption.

You should ask the person who created this API to see if there is or should be a way to get just alive workers status. Is this info not in the response?

srowen · 2015-06-26T11:02:15Z

Why do the filtering so many times?

I can refactor this better later. Let me find the API author first. Thanks

thegiive · 2015-06-26T15:17:19Z

HI Sean, thanks for your opinion.

My feeling is that WebUI and API is just different presentation format. But the content should be the same. Different meaning in the same column(workers,memory,core) will confuse the user.

BTW, Let me talk to API author first :)

thegiive · 2015-06-26T21:46:29Z

Base on git blame, I think the JSON API creator is @rxin ( 9db1e50 )

Hi Reynold

#6317 has already change some master UI's data's definition (total memory/memory in used , total core/core in used ) .
Base on the same reason list in #6317 and the WebUI should sync the number with JSON API.
I think it is reasonable to sync the JSON API's column definition.

How do you think?

andrewor14 · 2015-06-29T20:24:27Z

ok to test

andrewor14 · 2015-06-29T20:24:55Z

I think we should have the JSON protocol do whatever the normal UI does. This change looks fine.

andrewor14 · 2015-06-29T20:25:34Z

@thegiive have you verified that these are the only sources of mismatch?

AmplabJenkins · 2015-06-29T20:27:20Z

Merged build triggered.

AmplabJenkins · 2015-06-29T20:27:26Z

Merged build started.

SparkQA · 2015-06-29T20:28:35Z

Test build #36030 has started for PR 7038 at commit 431d2b0.

SparkQA · 2015-06-29T20:30:22Z

Test build #36030 has finished for PR 7038 at commit 431d2b0.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-06-29T20:30:47Z

Merged build finished. Test PASSed.

andrewor14 · 2015-07-02T06:17:44Z

Hm I just saw @srowen's comment. Actually this change is not backward compatible so I'm not sure if we can merge it. @rxin any thoughts?

rxin · 2015-07-02T06:23:41Z

what's the non-compatible part?

srowen · 2015-07-02T07:01:36Z

I think just that the response will have different information now -- are people relying on and properly expecting to get info for all workers in the JSON? then removing non-alive worker info is incompatible in that sense.

rxin · 2015-07-02T07:03:45Z

ic - to me it is almost incorrect in the past, and we are fixing a bug here ...

AmplabJenkins · 2015-07-03T02:38:11Z

Merged build triggered.

AmplabJenkins · 2015-07-03T02:38:21Z

Merged build started.

thegiive · 2015-07-03T02:38:33Z

Hi all reviewer

I modified the code base on the comment.
Can you check if it is ok ?

SparkQA · 2015-07-03T02:38:51Z

Test build #36464 has started for PR 7038 at commit 2c8ea89.

SparkQA · 2015-07-03T04:23:56Z

Test build #36464 has finished for PR 7038 at commit 2c8ea89.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-07-03T04:24:13Z

Merged build finished. Test FAILed.

andrewor14 · 2015-07-03T16:43:56Z

in Spark we generally use camel case, so:
val aliveWorkers = obj.workers.filter(_.state == WorkerState.ALIVE)

Hi Andrew
WorkerState is org.apache.spark.deploy.master's private object. JsonProtocol cannot resolved WorkerState due to class visibility.This code cannot work unless we change WorkerState code.
I think best way to do that is add a one line method on WorkerInfo
How do you think?

BTW, I will change the variable name to camel case without any problem

AmplabJenkins · 2015-07-03T21:48:10Z

Merged build triggered.

AmplabJenkins · 2015-07-03T21:48:16Z

Merged build started.

SparkQA · 2015-07-03T21:51:13Z

Test build #36515 has started for PR 7038 at commit 9e54bf0.

SparkQA · 2015-07-03T23:57:00Z

Test build #36515 has finished for PR 7038 at commit 9e54bf0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-07-03T23:57:35Z

Merged build finished. Test PASSed.

thegiive · 2015-07-04T01:48:26Z

HI @andrewor14

The test is passed. If there is any problem I need to change, please let me know. Thanks

srowen · 2015-07-06T08:00:31Z

LGTM

andrewor14 · 2015-07-06T18:46:46Z

Ok, I'm merging this into master. Thanks @thegiive.

[SPARK-8656] Fix the webUI and JSON API number is not synced
8b3b8e8

srowen reviewed Jun 26, 2015
View reviewed changes

Worker List should contain DEAD node also
431d2b0

andrewor14 mentioned this pull request Jun 30, 2015
[SPARK-8611] [WebUI] Fix wrong REST UI Protocol on standalone cluster Web UI Page #7006
Closed

Change some styling and add local variable
2c8ea89

andrewor14 reviewed Jul 3, 2015
View reviewed changes

Change variable name to camel case
9e54bf0

asfgit closed this in 9ff2033Jul 6, 2015

Uh oh!

Conversation

thegiive commented Jun 26, 2015

Uh oh!

AmplabJenkins commented Jun 26, 2015

Uh oh!

srowen commented Jun 26, 2015

Uh oh!

srowenJun 26, 2015

Choose a reason for hiding this comment

Uh oh!

thegiiveJun 26, 2015

Choose a reason for hiding this comment

Uh oh!

thegiive commented Jun 26, 2015

Uh oh!

thegiive commented Jun 26, 2015

Uh oh!

andrewor14 commented Jun 29, 2015

Uh oh!

andrewor14 commented Jun 29, 2015

Uh oh!

andrewor14 commented Jun 29, 2015

Uh oh!

AmplabJenkins commented Jun 29, 2015

Uh oh!

AmplabJenkins commented Jun 29, 2015

Uh oh!

SparkQA commented Jun 29, 2015

Uh oh!

SparkQA commented Jun 29, 2015

Uh oh!

AmplabJenkins commented Jun 29, 2015

Uh oh!

andrewor14 commented Jul 2, 2015

Uh oh!

rxin commented Jul 2, 2015

Uh oh!

srowen commented Jul 2, 2015

Uh oh!

rxin commented Jul 2, 2015

Uh oh!

AmplabJenkins commented Jul 3, 2015

Uh oh!

AmplabJenkins commented Jul 3, 2015

Uh oh!

thegiive commented Jul 3, 2015

Uh oh!

SparkQA commented Jul 3, 2015

Uh oh!

SparkQA commented Jul 3, 2015

Uh oh!

AmplabJenkins commented Jul 3, 2015

Uh oh!

andrewor14Jul 3, 2015

Choose a reason for hiding this comment

Uh oh!

thegiiveJul 3, 2015

Choose a reason for hiding this comment

Uh oh!

thegiiveJul 3, 2015

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Jul 3, 2015

Uh oh!

AmplabJenkins commented Jul 3, 2015

Uh oh!

SparkQA commented Jul 3, 2015

Uh oh!

SparkQA commented Jul 3, 2015

Uh oh!

AmplabJenkins commented Jul 3, 2015

Uh oh!

thegiive commented Jul 4, 2015

Uh oh!

srowen commented Jul 6, 2015

Uh oh!

andrewor14 commented Jul 6, 2015

Uh oh!

Reviewers