[SPARK-18364][YARN] Expose metrics for YarnShuffleService by ash211 · Pull Request #17401 · apache/spark

ash211 · 2017-03-23T20:43:43Z

Registers the shuffle server's metrics with the Hadoop Node Manager's
DefaultMetricsSystem.

What changes were proposed in this pull request?

Expose the shuffle service metrics not only on the ExternalShuffleService as done in SPARK-16405, but also in the YarnShuffleService. Because the YARN Node Manager's metrics system is the Hadoop-internal one and not the Dropwizard Metrics system that Spark uses, this requires a conversion from DW metrics to Hadoop metrics.

How was this patch tested?

Manual deploy of new yarn-shuffle jar into a Node Manager and verifying that the metrics appear in the Node Manager-standard location. This is JMX with an query endpoint at /jmx.

Resulting metrics look like this:

[user@host ~]$ curl -sk -XGET https://`hostname -f`:8042/jmx | jq . | grep 'shuffleservice' -B 1 -A 18
{
"name": "Hadoop:service=NodeManager,name=shuffleservice",
"modelerType": "shuffleservice",
"tag.Hostname": "<redacted>",
"openBlockRequestLatencyMillis_count": 1,
"openBlockRequestLatencyMillis_rate15": 0.0011080303990206543,
"openBlockRequestLatencyMillis_rate5": 0.0033057092356765017,
"openBlockRequestLatencyMillis_rate1": 0.015991117074135343,
"openBlockRequestLatencyMillis_rateMean": 0.003843993699021382,
"blockTransferRateBytes_count": 118,
"blockTransferRateBytes_rate15": 0.1307475870844372,
"blockTransferRateBytes_rate5": 0.39007368980982715,
"blockTransferRateBytes_rate1": 1.8869518147479705,
"blockTransferRateBytes_rateMean": 0.45359183094454836,
"registeredExecutorsSize": 2,
"registerExecutorRequestLatencyMillis_count": 2,
"registerExecutorRequestLatencyMillis_rate15": 0.001697343764758814,
"registerExecutorRequestLatencyMillis_rate5": 0.002970701813078509,
"registerExecutorRequestLatencyMillis_rate1": 0.0005857750515146702,
"registerExecutorRequestLatencyMillis_rateMean": 0.007687995987242345
},
[user@host ~]$

Registers the shuffle server's metrics with the Hadoop Node Manager's DefaultMetricsSystem.

SparkQA · 2017-03-23T21:02:27Z

Test build #75115 has finished for PR 17401 at commit 4caf4d2.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class YarnShuffleServiceMetrics implements MetricsSource

jerryshao · 2017-03-24T04:44:31Z

Actually we follow two space indent for java code in Spark, would you please change the format?

ash211 · 2017-03-24T05:29:38Z

Thanks for taking a look @jerryshao ! I've reformatted to two-space indentation and run ./dev/lint-java to make sure this code passes the linter.

src/main/java/org/apache/spark/sql/streaming/GroupStateTimeout.java still has some errors but those are separate from this change

jerryshao · 2017-03-24T05:39:10Z

+ * @param all if true, return all metrics even if unchanged.
+ */
+ @Override
+ public void getMetrics(MetricsCollector collector, boolean all) {


Is it possible to add a unit test to verify the correctness of converting codahale metrics to Hadoop metrics?

should be able to, I'm working on creating one now. By correctness, I think you mostly mean that the values passed through are the same, even though the naming schemes are different?

Yes, you're right. From my understanding the correctness means Gauge still coverts to Gauge, Meter still to Meter, not sure can it be guaranteed?

jerryshao · 2017-03-24T05:41:33Z

+ Method registerSourceMethod = metricsSystem.getClass().getDeclaredMethod("registerSource",
+ String.class, String.class, MetricsSource.class);
+ registerSourceMethod.setAccessible(true);
+ registerSourceMethod.invoke(metricsSystem, "shuffleservice", "Metrics on the Spark " +


nit. "shuffleService" camel case might be better.

will change shortly

SparkQA · 2017-03-24T05:48:54Z

Test build #75150 has finished for PR 17401 at commit 13aa4ff.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ash211 · 2017-03-24T07:15:33Z

Thanks again for the comments @jerryshao ! I've now added some tests to verify that the metrics get converted in the expected way to the collector, and camel-cased shuffleService

SparkQA · 2017-03-24T07:19:33Z

Test build #75157 has finished for PR 17401 at commit 9992c10.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-24T19:30:59Z

Test build #75165 has finished for PR 17401 at commit 96a0882.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ash211 · 2017-03-24T21:20:40Z

Ready for further review.

jerryshao · 2017-03-27T08:56:55Z

+ }
+
+ @VisibleForTesting
+ public static void collectMetric(MetricsRecordBuilder metricsRecordBuilder, String name, Metric metric) {


Is it necessary to use static here? Looks like here it is it is only for the test convenience.

I use static here to make it clear that the method does not need to be run in the context of an instance. This prevents it from accidentally accessing instance variables when I don't intend it to

jerryshao · 2017-03-27T08:58:18Z

+import org.mockito.Matchers._
+import org.mockito.Mockito.{mock, times, verify, when}
+import org.scalatest.Matchers
+import scala.collection.JavaConverters._


The import ordering is not correct, Scala package should be before the third party packages.

jerryshao · 2017-03-27T09:00:12Z

+ MetricsSystemImpl metricsSystem = (MetricsSystemImpl) DefaultMetricsSystem.instance();
+
+ Method registerSourceMethod = metricsSystem.getClass().getDeclaredMethod("registerSource",
+ String.class, String.class, MetricsSource.class);


Nit, here I think should be 2 space indent follow the above line.

jerryshao · 2017-03-27T09:15:07Z

+ String.class, String.class, MetricsSource.class);
+ registerSourceMethod.setAccessible(true);
+ registerSourceMethod.invoke(metricsSystem, "shuffleService", "Metrics on the Spark " +
+ "Shuffle Service", serviceMetrics);


With import order specified at http://spark.apache.org/contributing.html

SparkQA · 2017-03-28T19:56:51Z

Test build #75322 has finished for PR 17401 at commit 7c7d6d4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ash211 · 2017-03-28T23:35:27Z

@jerryshao ready for re-review

jerryshao

Sorry for the delay, just some style related comments. We should ping others to take a review.

jerryshao · 2017-04-05T13:26:12Z

 * A simple class to wrap all shuffle service wrapper metrics
 */
- private class ShuffleMetrics implements MetricSet {
+ @VisibleForTesting


@VisibleForTesting causes classpath issues. Please note this in the java doc instead (SPARK-11615).
This is a scalastyle output, would be better to remove this annotation.

jerryshao · 2017-04-05T13:28:46Z

 String authEnabledString = authEnabled ? "enabled" : "not enabled";
 logger.info("Started YARN shuffle service for Spark on port {}. " +
- "Authentication is {}. Registered executor file is {}", port, authEnabledString,
+"Authentication is {}. Registered executor file is {}", port, authEnabledString,


Nit: this 2 space indent looks like not necessary.

jerryshao · 2017-04-05T13:33:05Z

+ }
+ }
+
+ @VisibleForTesting


jerryshao · 2017-04-05T13:41:19Z

+ Gauge m = (Gauge) metric;
+ Object gaugeValue = m.getValue();
+ if (gaugeValue instanceof Integer) {
+ Integer intValue = (Integer) gaugeValue;


Does it mean that we could only handle integer Gauge, what if later on we add different metric in ExternalShuffleBlockHandler? Looks like we should also manually handle the case one by one here. At least we should an else branch for the fallback check.

HyukjinKwon · 2017-06-19T04:54:10Z

gentle ping @ash211. I just wonder if it is active now.

## What changes were proposed in this pull request? This PR is follow-up of closedapache#17401 which only ended due to of inactivity, but its still nice feature to have. Given review by jerryshao taken in consideration and edited: - VisibleForTesting deleted because of dependency conflicts - removed unnecessary reflection for `MetricsSystemImpl` - added more available types for gauge ## How was this patch tested? Manual deploy of new yarn-shuffle jar into a Node Manager and verifying that the metrics appear in the Node Manager-standard location. This is JMX with an query endpoint running on `hostname:port` Resulting metrics look like this: ``` curl -sk -XGET hostname:port | grep -v '#' | grep 'shuffleService' hadoop_nodemanager_openblockrequestlatencymillis_rate15{name="shuffleService",} 0.31428910657834713 hadoop_nodemanager_blocktransferratebytes_rate15{name="shuffleService",} 566144.9983653595 hadoop_nodemanager_blocktransferratebytes_ratemean{name="shuffleService",} 2464409.9678099006 hadoop_nodemanager_openblockrequestlatencymillis_rate1{name="shuffleService",} 1.2893844732240272 hadoop_nodemanager_registeredexecutorssize{name="shuffleService",} 2.0 hadoop_nodemanager_openblockrequestlatencymillis_ratemean{name="shuffleService",} 1.255574678369966 hadoop_nodemanager_openblockrequestlatencymillis_count{name="shuffleService",} 315.0 hadoop_nodemanager_openblockrequestlatencymillis_rate5{name="shuffleService",} 0.7661929192569739 hadoop_nodemanager_registerexecutorrequestlatencymillis_ratemean{name="shuffleService",} 0.0 hadoop_nodemanager_registerexecutorrequestlatencymillis_count{name="shuffleService",} 0.0 hadoop_nodemanager_registerexecutorrequestlatencymillis_rate1{name="shuffleService",} 0.0 hadoop_nodemanager_registerexecutorrequestlatencymillis_rate5{name="shuffleService",} 0.0 hadoop_nodemanager_blocktransferratebytes_count{name="shuffleService",} 6.18271213E8 hadoop_nodemanager_registerexecutorrequestlatencymillis_rate15{name="shuffleService",} 0.0 hadoop_nodemanager_blocktransferratebytes_rate5{name="shuffleService",} 1154114.4881816586 hadoop_nodemanager_blocktransferratebytes_rate1{name="shuffleService",} 574745.0749848988 ``` Closesapache#22485 from mareksimunek/SPARK-18364. Lead-authored-by: marek.simunek <marek.simunek@firma.seznam.cz> Co-authored-by: Andrew Ash <andrew@andrewash.com> Signed-off-by: Thomas Graves <tgraves@apache.org>

## What changes were proposed in this pull request? This PR is follow-up of closedapache#17401 which only ended due to of inactivity, but its still nice feature to have. Given review by jerryshao taken in consideration and edited: - VisibleForTesting deleted because of dependency conflicts - removed unnecessary reflection for `MetricsSystemImpl` - added more available types for gauge ## How was this patch tested? Manual deploy of new yarn-shuffle jar into a Node Manager and verifying that the metrics appear in the Node Manager-standard location. This is JMX with an query endpoint running on `hostname:port` Resulting metrics look like this: ``` curl -sk -XGET hostname:port | grep -v '#' | grep 'shuffleService' hadoop_nodemanager_openblockrequestlatencymillis_rate15{name="shuffleService",} 0.31428910657834713 hadoop_nodemanager_blocktransferratebytes_rate15{name="shuffleService",} 566144.9983653595 hadoop_nodemanager_blocktransferratebytes_ratemean{name="shuffleService",} 2464409.9678099006 hadoop_nodemanager_openblockrequestlatencymillis_rate1{name="shuffleService",} 1.2893844732240272 hadoop_nodemanager_registeredexecutorssize{name="shuffleService",} 2.0 hadoop_nodemanager_openblockrequestlatencymillis_ratemean{name="shuffleService",} 1.255574678369966 hadoop_nodemanager_openblockrequestlatencymillis_count{name="shuffleService",} 315.0 hadoop_nodemanager_openblockrequestlatencymillis_rate5{name="shuffleService",} 0.7661929192569739 hadoop_nodemanager_registerexecutorrequestlatencymillis_ratemean{name="shuffleService",} 0.0 hadoop_nodemanager_registerexecutorrequestlatencymillis_count{name="shuffleService",} 0.0 hadoop_nodemanager_registerexecutorrequestlatencymillis_rate1{name="shuffleService",} 0.0 hadoop_nodemanager_registerexecutorrequestlatencymillis_rate5{name="shuffleService",} 0.0 hadoop_nodemanager_blocktransferratebytes_count{name="shuffleService",} 6.18271213E8 hadoop_nodemanager_registerexecutorrequestlatencymillis_rate15{name="shuffleService",} 0.0 hadoop_nodemanager_blocktransferratebytes_rate5{name="shuffleService",} 1154114.4881816586 hadoop_nodemanager_blocktransferratebytes_rate1{name="shuffleService",} 574745.0749848988 ``` Closesapache#22485 from mareksimunek/SPARK-18364. Lead-authored-by: marek.simunek <marek.simunek@firma.seznam.cz> Co-authored-by: Andrew Ash <andrew@andrewash.com> Signed-off-by: Thomas Graves <tgraves@apache.org> (cherry picked from commit a802c69)

## What changes were proposed in this pull request? This PR proposes to close stale PRs, mostly the same instances with apache#18017Closesapache#14085 - [SPARK-16408][SQL] SparkSQL Added file get Exception: is a directory … Closesapache#14239 - [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism to accelerate shuffle stage. Closesapache#14567 - [SPARK-16992][PYSPARK] Python Pep8 formatting and import reorganisation Closesapache#14579 - [SPARK-16921][PYSPARK] RDD/DataFrame persist()/cache() should return Python context managers Closesapache#14601 - [SPARK-13979][Core] Killed executor is re spawned without AWS key… Closesapache#14830 - [SPARK-16992][PYSPARK][DOCS] import sort and autopep8 on Pyspark examples Closesapache#14963 - [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in lint-python Closesapache#15227 - [SPARK-17655][SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage Closesapache#15240 - [SPARK-17556] [CORE] [SQL] Executor side broadcast for broadcast joins Closesapache#15405 - [SPARK-15917][CORE] Added support for number of executors in Standalone [WIP] Closesapache#16099 - [SPARK-18665][SQL] set statement state to "ERROR" after user cancel job Closesapache#16445 - [SPARK-19043][SQL]Make SparkSQLSessionManager more configurable Closesapache#16618 - [SPARK-14409][ML][WIP] Add RankingEvaluator Closesapache#16766 - [SPARK-19426][SQL] Custom coalesce for Dataset Closesapache#16832 - [SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns Closesapache#17052 - [SPARK-19690][SS] Join a streaming DataFrame with a batch DataFrame which has an aggregation may not work Closesapache#17267 - [SPARK-19926][PYSPARK] Make pyspark exception more user-friendly Closesapache#17371 - [SPARK-19903][PYSPARK][SS] window operator miss the `watermark` metadata of time column Closesapache#17401 - [SPARK-18364][YARN] Expose metrics for YarnShuffleService Closesapache#17519 - [SPARK-15352][Doc] follow-up: add configuration docs for topology-aware block replication Closesapache#17530 - [SPARK-5158] Access kerberized HDFS from Spark standalone Closesapache#17854 - [SPARK-20564][Deploy] Reduce massive executor failures when executor count is large (>2000) Closesapache#17979 - [SPARK-19320][MESOS][WIP]allow specifying a hard limit on number of gpus required in each spark executor when running on mesos Closesapache#18127 - [SPARK-6628][SQL][Branch-2.1] Fix ClassCastException when executing sql statement 'insert into' on hbase table Closesapache#18236 - [SPARK-21015] Check field name is not null and empty in GenericRowWit… Closesapache#18269 - [SPARK-21056][SQL] Use at most one spark job to list files in InMemoryFileIndex Closesapache#18328 - [SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable Closesapache#18354 - [SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting Closesapache#18383 - [SPARK-21167][SS] Set kafka clientId while fetch messages Closesapache#18414 - [SPARK-21169] [core] Make sure to update application status to RUNNING if executors are accepted and RUNNING after recovery Closesapache#18432 - resolve com.esotericsoftware.kryo.KryoException Closesapache#18490 - [SPARK-21269][Core][WIP] Fix FetchFailedException when enable maxReqSizeShuffleToMem and KryoSerializer Closesapache#18585 - SPARK-21359 Closesapache#18609 - Spark SQL merge small files to big files Update InsertIntoHiveTable.scala Added: Closesapache#18308 - [SPARK-21099][Spark Core] INFO Log Message Using Incorrect Executor I… Closesapache#18599 - [SPARK-21372] spark writes one log file even I set the number of spark_rotate_log to 0 Closesapache#18619 - [SPARK-21397][BUILD]Maven shade plugin adding dependency-reduced-pom.xml to … Closesapache#18667 - Fix the simpleString used in error messages Closesapache#18782 - Branch 2.1 Added: Closesapache#17694 - [SPARK-12717][PYSPARK] Resolving race condition with pyspark broadcasts when using multiple threads Added: Closesapache#16456 - [SPARK-18994] clean up the local directories for application in future by annother thread Closesapache#18683 - [SPARK-21474][CORE] Make number of parallel fetches from a reducer configurable Closesapache#18690 - [SPARK-21334][CORE] Add metrics reporting service to External Shuffle Server Added: Closesapache#18827 - Merge pull request 1 from apache/master ## How was this patch tested? N/A Author: hyukjinkwon <gurwls223@gmail.com> Closesapache#18780 from HyukjinKwon/close-prs.

[SPARK-18364][YARN] Expose metrics for YarnShuffleService
4caf4d2
Registers the shuffle server's metrics with the Hadoop Node Manager's DefaultMetricsSystem.

2-space indentation on java files; pass ./dev/lint-java
13aa4ff

jerryshao reviewed Mar 24, 2017
View reviewed changes

ash211 added 2 commits March 24, 2017 00:13

Test metric collector gets right converted calls
83b1d73

camel-case shuffleService
9992c10

Pass scalastyle
96a0882

jerryshao reviewed Mar 27, 2017
View reviewed changes

Reformat and organize imports
7c7d6d4
With import order specified at http://spark.apache.org/contributing.html

jerryshao reviewed Apr 5, 2017
View reviewed changes

HyukjinKwon mentioned this pull request Jun 25, 2017
[INFRA] Close stale PRs #18417
Closed

HyukjinKwon mentioned this pull request Jul 31, 2017
[INFRA] Close stale PRs #18780
Closed

asfgit closed this in 3a45c7fAug 5, 2017

mareksimunek mentioned this pull request Sep 20, 2018
[SPARK-18364][YARN] Expose metrics for YarnShuffleService #22485
Closed

Uh oh!

Conversation

ash211 commented Mar 23, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Mar 23, 2017

Uh oh!

jerryshao commented Mar 24, 2017

Uh oh!

ash211 commented Mar 24, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 24, 2017

Uh oh!

ash211 commented Mar 24, 2017

Uh oh!

SparkQA commented Mar 24, 2017

Uh oh!

SparkQA commented Mar 24, 2017

Uh oh!

ash211 commented Mar 24, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jerryshaoMar 27, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 28, 2017

Uh oh!

ash211 commented Mar 28, 2017

Uh oh!

jerryshao left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Jun 19, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ash211 commented Mar 23, 2017•
edited
Loading

jerryshaoMar 27, 2017•
edited
Loading