Skip to content

[SPARK-19824][Core] Standalone master JSON not showing cores for running applications - #17181

Closed
yongtang wants to merge 1 commit into
apache:masterfrom
yongtang:SPARK-19824
Closed

[SPARK-19824][Core] Standalone master JSON not showing cores for running applications#17181
yongtang wants to merge 1 commit into
apache:masterfrom
yongtang:SPARK-19824

Conversation

@yongtang

@yongtangyongtang commented Mar 6, 2017

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This fix tries to address the issue raised in SPARK-19824 where
the JSON API of the standalone master ("/json") does not show
the number of cores for a running (active) application, which is
available on the master UI.

The reason was that in master UI, coresGranted was used:
https://github.com/apache/spark/blob/v2.1.0/core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala#L220
while in JSON API, the desc.maxCores was used. In case
there is no limit on maxCores, this filed will be missing on
JSON API. (Master UI show cores granted so the field is always available).

This fix address the discrepancy by changing JSON API to use
coredGranted so that UI and JSON API are consistent.

How was this patch tested?

The test was done manually.

Please review http://spark.apache.org/contributing.html before opening a pull request.

@yongtang
[SPARK-19824][Core] Standalone master JSON not showing cores for runn…
f8b5eaf
…ing applications
This fix tries to address the issue raised in SPARK-19824 where the
JSON API of the standalone master ("/json") does not show the number of
cores for a running (active) application, which is available on the
master UI.
The reason was that in master UI, `coresGranted` was used:
https://github.com/apache/spark/blob/v2.1.0/core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala#L220
while in JSON API, the `desc.maxCores` was used. In case there is no limit on
`maxCores`, this filed will be missing on JSON API. (Master UI show cores granted
so the field is always available).
This fix address the discrepancy by changing JSON API to use `coredGranted`
so that UI and JSON API are consistent.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
nanop
nanop reviewed Mar 6, 2017
View reviewed changes
Comment threadcore/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala
("starttime" -> obj.startTime) ~
("id" -> obj.id) ~
("name" -> obj.desc.name) ~
("cores" -> obj.desc.maxCores) ~

@nanopnanopMar 6, 2017

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not keep both?

@yongtangyongtangMar 6, 2017

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nanop this is to align with Master UI. I can update the PR to add additional field in the JSON API if confirmed from maintainers.

@jiangxb1987

jiangxb1987 commented May 25, 2017•
edited
Loading

Copy link
Copy Markdown
Contributor

I think the fix is correct, it would be great if you could add test cases to cover this in JsonProtocolSuite.
Also cc @gatorsmile@cloud-fan

@cloud-fan

cloud-fan commented May 25, 2017

Copy link
Copy Markdown
Contributor

yea, also a screenshot about this in Spark UI will be great

@gatorsmile

gatorsmile commented May 25, 2017

Copy link
Copy Markdown
Member

ok to test

@gatorsmile

gatorsmile commented May 25, 2017•
edited
Loading

Copy link
Copy Markdown
Member

Could you please check whether there exists any other inconsistent value between the UI and JSON API? Thanks!

@SparkQA

SparkQA commented May 25, 2017

Copy link
Copy Markdown

Test build #77335 has finished for PR 17181 at commit f8b5eaf.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jiangxb1987

jiangxb1987 commented May 26, 2017

Copy link
Copy Markdown
Contributor

ping @yongtang

@jiangxb1987jiangxb1987 mentioned this pull request Jun 14, 2017
Closed
asfgit pushed a commit that referenced this pull request Jun 19, 2017
@jiangxb1987@gatorsmile
[SPARK-19824][CORE] Update JsonProtocol to keep consistent with the UI
ea542d2
## What changes were proposed in this pull request?
Fix any inconsistent part in JsonProtocol with the UI.
This PR also contains the modifications in #17181
## How was this patch tested?
Updated JsonProtocolSuite.
Before this change, localhost:8080/json shows:
```
{
"url" : "spark://xingbos-MBP.local:7077",
"workers" : [ {
"id" : "worker-20170615172946-192.168.0.101-49450",
"host" : "192.168.0.101",
"port" : 49450,
"webuiaddress" : "http://192.168.0.101:8081",
"cores" : 8,
"coresused" : 8,
"coresfree" : 0,
"memory" : 15360,
"memoryused" : 1024,
"memoryfree" : 14336,
"state" : "ALIVE",
"lastheartbeat" : 1497519481722
}, {
"id" : "worker-20170615172948-192.168.0.101-49452",
"host" : "192.168.0.101",
"port" : 49452,
"webuiaddress" : "http://192.168.0.101:8082",
"cores" : 8,
"coresused" : 8,
"coresfree" : 0,
"memory" : 15360,
"memoryused" : 1024,
"memoryfree" : 14336,
"state" : "ALIVE",
"lastheartbeat" : 1497519484160
}, {
"id" : "worker-20170615172951-192.168.0.101-49469",
"host" : "192.168.0.101",
"port" : 49469,
"webuiaddress" : "http://192.168.0.101:8083",
"cores" : 8,
"coresused" : 8,
"coresfree" : 0,
"memory" : 15360,
"memoryused" : 1024,
"memoryfree" : 14336,
"state" : "ALIVE",
"lastheartbeat" : 1497519486905
} ],
"cores" : 24,
"coresused" : 24,
"memory" : 46080,
"memoryused" : 3072,
"activeapps" : [ {
"starttime" : 1497519426990,
"id" : "app-20170615173706-0001",
"name" : "Spark shell",
"user" : "xingbojiang",
"memoryperslave" : 1024,
"submitdate" : "Thu Jun 15 17:37:06 CST 2017",
"state" : "RUNNING",
"duration" : 65362
} ],
"completedapps" : [ {
"starttime" : 1497519250893,
"id" : "app-20170615173410-0000",
"name" : "Spark shell",
"user" : "xingbojiang",
"memoryperslave" : 1024,
"submitdate" : "Thu Jun 15 17:34:10 CST 2017",
"state" : "FINISHED",
"duration" : 116895
} ],
"activedrivers" : [ ],
"status" : "ALIVE"
}
```
After the change:
```
{
"url" : "spark://xingbos-MBP.local:7077",
"workers" : [ {
"id" : "worker-20170615175032-192.168.0.101-49951",
"host" : "192.168.0.101",
"port" : 49951,
"webuiaddress" : "http://192.168.0.101:8081",
"cores" : 8,
"coresused" : 8,
"coresfree" : 0,
"memory" : 15360,
"memoryused" : 1024,
"memoryfree" : 14336,
"state" : "ALIVE",
"lastheartbeat" : 1497520292900
}, {
"id" : "worker-20170615175034-192.168.0.101-49953",
"host" : "192.168.0.101",
"port" : 49953,
"webuiaddress" : "http://192.168.0.101:8082",
"cores" : 8,
"coresused" : 8,
"coresfree" : 0,
"memory" : 15360,
"memoryused" : 1024,
"memoryfree" : 14336,
"state" : "ALIVE",
"lastheartbeat" : 1497520280301
}, {
"id" : "worker-20170615175037-192.168.0.101-49955",
"host" : "192.168.0.101",
"port" : 49955,
"webuiaddress" : "http://192.168.0.101:8083",
"cores" : 8,
"coresused" : 8,
"coresfree" : 0,
"memory" : 15360,
"memoryused" : 1024,
"memoryfree" : 14336,
"state" : "ALIVE",
"lastheartbeat" : 1497520282884
} ],
"aliveworkers" : 3,
"cores" : 24,
"coresused" : 24,
"memory" : 46080,
"memoryused" : 3072,
"activeapps" : [ {
"id" : "app-20170615175122-0001",
"starttime" : 1497520282115,
"name" : "Spark shell",
"cores" : 24,
"user" : "xingbojiang",
"memoryperslave" : 1024,
"submitdate" : "Thu Jun 15 17:51:22 CST 2017",
"state" : "RUNNING",
"duration" : 10805
} ],
"completedapps" : [ {
"id" : "app-20170615175058-0000",
"starttime" : 1497520258766,
"name" : "Spark shell",
"cores" : 24,
"user" : "xingbojiang",
"memoryperslave" : 1024,
"submitdate" : "Thu Jun 15 17:50:58 CST 2017",
"state" : "FINISHED",
"duration" : 9876
} ],
"activedrivers" : [ ],
"completeddrivers" : [ ],
"status" : "ALIVE"
}
```
Author: Xingbo Jiang <xingbo.jiang@databricks.com>
Closes#18303 from jiangxb1987/json-protocol.
@HyukjinKwonHyukjinKwon mentioned this pull request Jun 25, 2017
Closed
@gatorsmile

gatorsmile commented Jun 27, 2017

Copy link
Copy Markdown
Member

We are closing it due to inactivity. please do reopen if you want to push it forward. Thanks!

@yongtang
yongtang deleted the SPARK-19824 branch January 19, 2018 01:04
zifeif2 pushed a commit to zifeif2/spark that referenced this pull request Nov 22, 2025
@HyukjinKwon@zifeif2
[INFRA] Close stale PRs
ddbe288
## What changes were proposed in this pull request?
This PR proposes to close stale PRs, mostly the same instances with apache#18017
I believe the author in apache#14807 removed his account.
Closesapache#7075Closesapache#8927Closesapache#9202Closesapache#9366Closesapache#10861Closesapache#11420Closesapache#12356Closesapache#13028Closesapache#13506Closesapache#14191Closesapache#14198Closesapache#14330Closesapache#14807Closesapache#15839Closesapache#16225Closesapache#16685Closesapache#16692Closesapache#16995Closesapache#17181Closesapache#17211Closesapache#17235Closesapache#17237Closesapache#17248Closesapache#17341Closesapache#17708Closesapache#17716Closesapache#17721Closesapache#17937
Added:
Closesapache#14739Closesapache#17139Closesapache#17445Closesapache#18042Closesapache#18359
Added:
Closesapache#16450Closesapache#16525Closesapache#17738
Added:
Closesapache#16458Closesapache#16508Closesapache#17714
Added:
Closesapache#17830Closesapache#14742
## How was this patch tested?
N/A
Author: hyukjinkwon <gurwls223@gmail.com>
Closesapache#18417 from HyukjinKwon/close-stale-pr.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@nanopnanopnanop left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@yongtang@jiangxb1987@cloud-fan@gatorsmile@SparkQA@nanop