TEZ-3331: Add operation specific HDFS counters for Tez UI by abstractdog · Pull Request #379 · apache/tez

abstractdog · 2024-11-07T10:27:14Z

This a rebased, reworked version of the last patch on TEZ-3331:
https://issues.apache.org/jira/secure/attachment/12926702/TEZ-3331.8.patch

abstractdog · 2024-11-21T07:20:00Z

@ayushtkn, can you please review this one? we're using this downstream for a long time, haven't ported back to upstream yet

ayushtkn

Thanx Laszlo for the PR, minor stuff rest looks good

ayushtkn · 2024-11-23T05:49:18Z

+ FILE_BYTES_READ("fileBytesRead"),
+ FILE_BYTES_WRITTEN("fileBytesWritten"),
+
+ // Additional counters from HADOOP-13305


There seems to be some more counters as well, like OP_CREATE_NON_RECURSIVE, OP_EXISTS, OP_IS_FILE etc
Should we include them as well?

yeah, I'm refreshing it according to the current state of CommonStatisticNames

ayushtkn · 2024-11-23T05:59:53Z

+ if (!statisticUpdaters.containsKey(stats.getScheme())) {
+ Map<String, FileSystemStatisticUpdater> updaterSet = new TreeMap<>();
+ statisticUpdaters.put(stats.getScheme(), updaterSet);
 }
+ FileSystemStatisticUpdater updater = statisticUpdaters.get(stats.getScheme())
+ .computeIfAbsent(stats.getName(), k -> new FileSystemStatisticUpdater(tezCounters, stats));


this looks like computeIfAbsent
maybe can do
// Fetch or initialize the updater set for the scheme Map<String, FileSystemStatisticUpdater> updaterSet = statisticUpdaters .computeIfAbsent(stats.getScheme(), k -> new TreeMap<>()); // Fetch or create the updater for the specific statistic FileSystemStatisticUpdater updater = updaterSet .computeIfAbsent(stats.getName(), k -> new FileSystemStatisticUpdater(tezCounters, stats));

absolutely, thanks!

ayushtkn · 2024-11-23T06:01:02Z

+
+ private static MiniDFSCluster dfsCluster;
+
+ private static Configuration conf = new Configuration();


can be final

ayushtkn · 2024-11-23T06:01:33Z

+ public static void setup() throws IOException {
+ try {
+ conf.set(MiniDFSCluster.HDFS_MINIDFS_BASEDIR, TEST_ROOT_DIR);
+ dfsCluster = new MiniDFSCluster.Builder(conf).numDataNodes(2).format(true).racks(null)


.format(true).racks(null) ain't required IMO, they are by default true & null

ayushtkn · 2024-11-23T06:04:35Z

+ FSDataOutputStream out = remoteFs.create(new Path("/tmp/foo/abc.txt"));
+ out.writeBytes("xyz");
+ out.close();


can use this
DFSTestUtil.writeFile(remoteFs, new Path("/tmp/foo/abc.txt"), "xyz");
and below as well

I can see there is a difference between how these approaches work, which is reflected in the counters
no matter which one we choose, the counters can be asserted accordingly
original approach (mkdir, create)
HDFS_BYTES_WRITTEN=3 HDFS_WRITE_OPS=2 HDFS_OP_CREATE=1 HDFS_OP_MKDIRS=1
DFSTestUtil.writeFile
HDFS_BYTES_WRITTEN=3 HDFS_READ_OPS=1 HDFS_WRITE_OPS=1 HDFS_OP_CREATE=1 HDFS_OP_GET_FILE_STATUS=1

ayushtkn · 2024-11-23T06:08:19Z

+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ TestTaskCounterUpdater.class);
+ private static Configuration conf = new Configuration();


can be final

ayushtkn · 2024-11-23T06:08:42Z

+ private static final Logger LOG = LoggerFactory.getLogger(
+ TestTaskCounterUpdater.class);


nit
line break wasn't required I think

ayushtkn · 2024-11-23T06:10:08Z

+ TaskCounterUpdater updater = new TaskCounterUpdater(counters, conf, "pid");
+
+ updater.updateCounters();
+ LOG.info("Counters: " + counters);


ayushtkn · 2024-11-23T06:10:53Z

+ Assert.assertTrue("Counter not updated, old=" + oldVal
+ + ", new=" + cpuCounter.getValue(), cpuCounter.getValue() > oldVal);


should we put some sleep before updateCounters, just thinking in extreme conditions, this check won't go flaky, right?

not sure, it seems quite unlikely, I would let it be decided by the future's precommit runs (it will be obvious once it fails)

ayushtkn · 2024-11-23T06:10:58Z

+
+

ayushtkn · 2024-11-23T06:16:13Z

+ OP_SET_ACL(CommonStatisticNames.OP_SET_ACL),
+ OP_SET_OWNER(CommonStatisticNames.OP_SET_OWNER),
+ OP_SET_PERMISSION(CommonStatisticNames.OP_SET_PERMISSION),
+ OP_GET_FILE_BLOCK_LOCATIONS("op_get_file_block_locations");


what is this, no enum in Hadoop? where is this coming from then?

it's from DFSOpsCountStatistics, but I'm removing it as I intend to include and expose only stats that are defined in CommonStatisticNames

abstractdog · 2024-11-23T10:53:12Z

thanks @ayushtkn for the initial comments! I need to rework this area, I think some of the comments are related to the fact that I simply adapted an old patch, I need to think this over again! I'll let you know

ayushtkn

Thanx @abstractdog for the changes, Minor Comments, rest looks good

ayushtkn · 2024-12-10T18:04:22Z

+ // Additional counters from HADOOP-13305
+ // Additional counters from HADOOP-13305


Duplicate line

ayushtkn · 2024-12-10T18:08:02Z

+// /**
+// * A Map where Key-> URIScheme and value->FileSystemStatisticUpdater
+// */
+// private Map<String, FileSystemStatisticUpdater> statisticUpdaters =
+// new HashMap<>();


why are you commenting this out? Can't we just delete it

oh right, leftover, removing it

ayushtkn · 2024-12-10T18:08:42Z

- private Map<String, FileSystemStatisticUpdater> statisticUpdaters =
- new HashMap<String, FileSystemStatisticUpdater>();
+ private Map<String, Map<String, FileSystemStatisticUpdater>> statisticUpdaters =
+new HashMap<>();


this we can make final

ayushtkn · 2024-12-10T18:12:02Z

+ StorageStatistics stats = iter.next();
+ // Fetch or initialize the updater set for the scheme
+ Map<String, FileSystemStatisticUpdater> updaterSet = statisticUpdaters
+ .computeIfAbsent(stats.getScheme(), k -> new TreeMap<>());


Why are we using TreeMap now? If I decode right, earlier it was HashMap, it would be some cost using it right?

hm, I cannot see the point of ordering while adding different statistic updaters for the same scheme, let me revert back to HashMap

ayushtkn · 2024-12-10T18:15:05Z

+
+ DFSTestUtil.writeFile(remoteFs, new Path("/tmp/foo/abc.txt"), "xyz");
+
+ updater.updateCounters();


shouldn't we first do
FileSystem.clearStatistics();
In case there is any test added in future which does FS operations, I believe this test will screw up. So better to reset everything to 0, before we start testing

taking care of that in a @before method
tested it with copying basicTest to basicTest2 and tried until the problems went away :)

abstractdog · 2024-12-12T11:21:13Z

Thanx @abstractdog for the changes, Minor Comments, rest looks good

thanks @ayushtkn, addressed your comments

tez-yetus · 2024-12-12T11:46:59Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 9s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 3 new or modified test files.
		_ master Compile Tests _
+0 🆗	mvndep	5m 46s	Maven dependency ordering for branch
+1 💚	mvninstall	7m 35s	master passed
+1 💚	compile	0m 34s	master passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚	compile	0m 38s	master passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~22.04-ga
+1 💚	checkstyle	1m 0s	master passed
+1 💚	javadoc	0m 50s	master passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	0m 38s	master passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~22.04-ga
+0 🆗	spotbugs	0m 29s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 38s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 7s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 29s	the patch passed
+1 💚	compile	0m 27s	the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚	javac	0m 27s	the patch passed
+1 💚	compile	0m 20s	the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~22.04-ga
+1 💚	javac	0m 20s	the patch passed
+1 💚	checkstyle	0m 5s	The patch passed checkstyle in tez-api
+1 💚	checkstyle	0m 6s	tez-runtime-internals: The patch generated 0 new + 16 unchanged - 1 fixed = 16 total (was 17)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+1 💚	javadoc	0m 22s	the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	0m 25s	the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~22.04-ga
+1 💚	findbugs	0m 56s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 51s	tez-api in the patch passed.
+1 💚	unit	0m 38s	tez-runtime-internals in the patch passed.
+1 💚	asflicense	0m 20s	The patch does not generate ASF License warnings.
		25m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-379/6/artifact/out/Dockerfile
GITHUB PR	#379
JIRA Issue	TEZ-3331
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile xml
uname	Linux 9b784bc2902b 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `b5bf8dc`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~22.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~22.04-ga
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-379/6/testReport/
Max. process+thread count	371 (vs. ulimit of 5500)
modules	C: tez-api tez-runtime-internals U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-379/6/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn

LGTM

abstractdog force-pushed the TEZ-3331 branch from a788e64 to 2a60cf5Compare November 7, 2024 10:29

This comment was marked as outdated.
Sign in to view

apache deleted a comment from tez-yetus Nov 7, 2024

abstractdog force-pushed the TEZ-3331 branch from 2a60cf5 to 60db8f1Compare November 7, 2024 11:31

This comment was marked as outdated.
Sign in to view

abstractdog requested a review from ayushtkn November 21, 2024 07:19

ayushtkn reviewed Nov 23, 2024
View reviewed changes

TEZ-3331: Add operation specific HDFS counters for Tez UI
c909e78

abstractdog force-pushed the TEZ-3331 branch from 60db8f1 to 532adecCompare November 24, 2024 00:04

This comment was marked as outdated.
Sign in to view

PR comments, further refactoring
d642287

abstractdog force-pushed the TEZ-3331 branch from 532adec to d642287Compare November 24, 2024 07:57

This comment was marked as outdated.
Sign in to view

ayushtkn reviewed Dec 10, 2024
View reviewed changes

PR comments 2
082e2b5

ayushtkn approved these changes Dec 12, 2024
View reviewed changes

abstractdog merged commit 1f7465f into apache:masterDec 13, 2024


		private static MiniDFSCluster dfsCluster;

		private static Configuration conf = new Configuration();

		private static final Logger LOG = LoggerFactory.getLogger(
		TestTaskCounterUpdater.class);

		Assert.assertTrue("Counter not updated, old=" + oldVal
		+ ", new=" + cpuCounter.getValue(), cpuCounter.getValue() > oldVal);

		// Additional counters from HADOOP-13305
		// Additional counters from HADOOP-13305


		DFSTestUtil.writeFile(remoteFs, new Path("/tmp/foo/abc.txt"), "xyz");

		updater.updateCounters();

Uh oh!

Conversation

abstractdog commented Nov 7, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

abstractdog commented Nov 21, 2024

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abstractdogNov 24, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abstractdog commented Nov 23, 2024

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abstractdogDec 12, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abstractdog commented Dec 12, 2024

Uh oh!

tez-yetus commented Dec 12, 2024

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

abstractdog commented Nov 7, 2024•
edited
Loading

abstractdogNov 24, 2024•
edited
Loading

abstractdogDec 12, 2024•
edited
Loading