HDDS-15678. OFS isDirectory/isFile should not trigger pipeline refresh or return block locations by yandrey321 · Pull Request #10741 · apache/ozone

yandrey321 · 2026-07-13T13:50:42Z

What changes were proposed in this pull request?

On OFS, isDirectory(path) and isFile(path) fall through to Hadoop's FileSystem defaults, which call getFileStatus(path).isDirectory(). For a file path this issues a full GetFileStatus OM RPC that, server-side, calls refresh() → refreshPipeline — an extra OM→SCM round-trip to repopulate pipeline/block-location info — and returns the full block-location list. A type check needs none of that.

This change makes the OFS type-check path a metadata-only (head-op) request:

Client plumbing. A headOp flag is threaded from BasicRootedOzoneFileSystem.isDirectory/isFile down through the adapter, OzoneBucket, ClientProtocol, and RpcClient into the OmKeyArgs used by getFileStatus. Existing callers of getFileStatus keep the full behaviour via overloads/default methods that pass headOp=false. This mirrors the existing RpcClient.getKeyInfo(..., isHeadOp) pattern used by S3 HEAD.
OM honours the flag. KeyManagerImpl already skips refresh + sortDatanodes when args.isHeadOp(); the change makes that flag actually arrive:

the client translator (OzoneManagerProtocolClientSideTranslatorPB.getFileStatus) now sets headOp on the request KeyArgs, and
the server handler (OzoneManagerRequestHandler.getOzoneFileStatus) now reads it back into the OmKeyArgs handed to the OM.

Block locations dropped for head ops. Because a head op skips the refresh, its block locations carry no pipeline (and are unused by a type check). The server handler strips keyLocationList from the head-op response proto (on an immutable toBuilder() copy — no OM-cache mutation), keeping the response small.

No protobuf change is required — KeyArgs.headOp (field 18) already exists — and the change is wire-compatible: an old server ignores the flag (full behaviour); an old client never sets it.

Side notes:

The flag was originally dropped by both the client translator and the server handler, so a client-only change was a no-op end-to-end — both sides are now fixed.

Skipping the refresh returns block locations with null pipelines; stripping them server-side (item 3) is what prevents a client-side NPE for multi-block files, so it is required, not just an optimization.

co-authored with Claude Code

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15678

How was this patch tested?

Added unit tests:
TestRpcClientGetFileStatusHeadOp — headOp reaches the wire KeyArgs (true/false).
TestRootedOzoneFileSystemHeadOp — isDirectory/isFile use head-op; correct results for file/dir/missing paths; error mapping.
TestBasicRootedOzoneClientAdapterHeadOp — adapter dispatch + headOp threading (key/root/volume/snapshot, OMException mapping).
TestOzoneBucket and TestOzoneManagerRequestHandler — bucket/protocol defaults and the handler forwarding headOp + stripping keyLocationList (retained when not head-op; defensive no-keyInfo path).

Integration in AbstractRootedOzoneFileSystemTest#testIsFileAndIsDirectory (runs on TestOFS, TestOFSWithFSO, TestOFSWithFSPaths).

No-regression run: unit tests for common, client, ozonefs-common, ozone-manager (clean rebuild — ozone-manager alone 2,530 tests) and the three OFS integration suites (181 tests) — all pass. checkstyle clean on all changed modules.

Benchmark (TestOFSIsDirectoryBenchmark, @tag("benchmark"), not in the normal suite): on a MiniOzoneCluster, isFile (head-op) vs full getFileStatus on a 1-block file — 1.47× throughput (5,078 vs 3,458 ops/s). The gain is understated here (in-process SCM RPC); over a real network the eliminated round-trip costs more.

Copilot

Pull request overview

This PR optimizes OFS isDirectory(path) / isFile(path) by introducing a metadata-only “headOp” getFileStatus path that avoids OM→SCM pipeline refresh and omits block locations from the response, while keeping full getFileStatus behavior unchanged for existing callers.

Changes:

Thread a headOp flag from BasicRootedOzoneFileSystem.isDirectory/isFile through OzoneClientAdapter → rooted adapter → OzoneBucket → ClientProtocol/RpcClient → OM PB translator into wire KeyArgs.
Update OzoneManagerRequestHandler to forward headOp into OmKeyArgs and strip keyLocationList from head-op GetFileStatus responses.
Add unit + integration coverage (and an on-demand benchmark) validating head-op propagation, error mapping, and correct type-check results.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/OzoneClientAdapter.java	Adds `getFileStatus(..., headOp)` default overload for metadata-only status requests.
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneFileSystem.java	Uses head-op for `isDirectory/isFile`; adds `getFileStatusAdapter(..., headOp)` overload.
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneClientAdapterImpl.java	Threads `headOp` down to `OzoneBucket.getFileStatus(key, headOp)`.
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolClientSideTranslatorPB.java	Sets `KeyArgs.headOp` on `GetFileStatus` requests.
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/protocol/ClientProtocol.java	Adds default `getOzoneFileStatus(..., headOp)` overload (fallback for impls that don’t honor it).
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java	Implements `getOzoneFileStatus(..., headOp)` and sets `OmKeyArgs.headOp`.
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java	Adds `getFileStatus(key, headOp)` API and forwards to `ClientProtocol`.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java	Forwards `headOp` into `OmKeyArgs` and strips block locations on head-op responses.
hadoop-ozone/ozonefs-common/src/test/java/org/apache/hadoop/fs/ozone/TestRootedOzoneFileSystemHeadOp.java	New unit tests validating OFS `isDirectory/isFile` use head-op and map missing paths correctly.
hadoop-ozone/ozonefs-common/src/test/java/org/apache/hadoop/fs/ozone/TestBasicRootedOzoneClientAdapterHeadOp.java	New unit tests validating rooted adapter dispatch + headOp threading and error mapping.
hadoop-ozone/client/src/test/java/org/apache/hadoop/ozone/client/TestRpcClientGetFileStatusHeadOp.java	New unit test ensuring `headOp` reaches wire `KeyArgs`.
hadoop-ozone/client/src/test/java/org/apache/hadoop/ozone/client/TestOzoneBucket.java	New unit tests validating bucket/protocol headOp forwarding + default delegation behavior.
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/protocolPB/TestOzoneManagerRequestHandler.java	Adds tests for headOp forwarding and response location stripping (and defensive no-keyInfo case).
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/AbstractRootedOzoneFileSystemTest.java	Adds integration coverage for correct type-check behavior on file/dir/missing paths.
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestOFSIsDirectoryBenchmark.java	Adds on-demand benchmark (tagged) comparing head-op type checks vs full `getFileStatus`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+ ArgumentCaptor<Boolean> headOp = ArgumentCaptor.forClass(Boolean.class);
+ verify(adapter, org.mockito.Mockito.atLeastOnce()).getFileStatus(
+ anyString(), any(URI.class), any(Path.class), anyString(),
+ headOp.capture());
+ assertTrue(headOp.getValue(), "isDirectory/isFile must request headOp");


jojochuang

This PR is partially valid because OM refreshes only if the object is a file. Still can be useful for isDirectory(). However FileSystem.isDirectory() is deprecated so not sure how many applications use it.

ozone/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java

Line 1686 in 0c19a9b

if (fileStatus.isFile()) {

jojochuang · 2026-07-15T17:32:38Z

+ * -Dsurefire.failIfNoSpecifiedTests=false
+ * </pre>
+ */
+@Tag("benchmark")


I wonder if we want a separate Maven module for the benchmark test. It would be useful to develop & collect benchmark tests somewhere, which are run on demand or before release.

jojochuang · 2026-07-15T18:33:37Z

I think other than the omitted headOp in ClientProtocol.getOzoneFileStatus(), the rest are mostly good.

As a follow up item,
We can check other usage of FileStatus.isDirectory() and FileStatus.isFile(), and if the file status is only used to check if its file or directory, update them to include headOp.

For example:

ozone/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java

Line 1902 in f41cd8e

if (prevStatus.isDirectory()) {

jojochuang

a few comments below.

yandrey321 · 2026-07-15T21:52:54Z

I think other than the omitted headOp in ClientProtocol.getOzoneFileStatus(), the rest are mostly good.
As a follow up item, We can check other usage of FileStatus.isDirectory() and FileStatus.isFile(), and if the file status is only used to check if its file or directory, update them to include headOp.
For example:
ozone/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java
Line 1902 in f41cd8e
if (prevStatus.isDirectory()) {

should be fixed now

jojochuang · 2026-07-15T22:17:10Z

LGTM

A few more for follow-up:

Candidate	Only type-checked?	Head-op API available?	Effort
BasicOzoneFileSystem.isDirectory/isFile (o3fs)	Yes	No — needs adapter plumbing	Medium (mirror the PR for o3fs)
DeleteKeyHandler L103, L125	Yes	Yes (OzoneBucket.getFileStatus(k, headOp))	Trivial
BasicOzoneFileSystem L787 (trash)	Yes	Only via getFileStatusAdapter(p,true)	Small
BasicRootedOzoneClientAdapterImpl L764 (trash)	First call yes (2nd keeps status)	Via adapter	Small

The two clearly worth doing are theo3fsisDirectory/isFile(consistency with this PR's whole point) andDeleteKeyHandler(free win, API already exists).

jojochuang · 2026-07-16T00:36:58Z

Merged. Thanks @yandrey321

* master: (53 commits) HDDS-15876. Remove ozone-csi (apache#10780) HDDS-15508. Add 2.2.0 to compatibility/upgrade test (apache#10782) HDDS-15867. [JDK25] Bump byte-buddy to 1.18.11 (apache#10776) HDDS-15809. Validate x-amz-copy-source-range on UploadPartCopy (apache#10711). HDDS-15678. OFS isDirectory/isFile should not trigger pipeline refresh or return block locations (apache#10741) HDDS-15171. Add available space check on follower during bootstrap. (apache#10185) HDDS-14043. Fix ls -e UnsupportedOperationException on ofs/o3fs (apache#10209) HDDS-15709. Replace random name suffix with uniqueObjectName (apache#10773) HDDS-15849. Force channel shutdown in XceiverClientGrpc.close (apache#10747) HDDS-15765. Fail fast when --node-id is omitted for OM compact/defrag on an HA cluster (apache#10699) HDDS-14989. Delay follower SCM DN server start until Ratis log catch-up. (apache#10617). HDDS-15456. Add SCM DB lookup and identify orphan(wrt SCM) and deleted-but-present containers (apache#10547) HDDS-15812. Rename test utils for naming convention (apache#10766) HDDS-15600. Fix ListObjects response for encoding-type and empty delimiter (apache#10586). HDDS-15656. Fix non-atomic data size accumulation in ContainerBalancer move callback (apache#10708) HDDS-15667. `containerbalancer status`: show stop reason and iteration details after stop (apache#10681) HDDS-15823. Fix flaky TestFailureHandlingByClient#testContainerExclusionWithClosedContainerException (apache#10721) HDDS-15837. Add health-aware container ID listing in SCM (apache#10736) HDDS-15851. Return 403 for expired or out-of-range pre-signed S3 URLs (apache#10749) HDDS-15836. Add Archiver.appendFile for incremental TAR writes (apache#10735) ... Conflicts: hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeMetrics.java hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyInfo.java hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/RepeatedOmKeyInfo.java hadoop-ozone/dist/src/main/compose/upgrade/test.sh hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/HddsUpgradeTestUtils.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/TestDNDataDistributionFinalization.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/TestHDDSUpgrade.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/TestScmDataDistributionFinalization.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/TestScmHAFinalization.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/service/TestBlockDeletionService.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java

* HDDS-14496-zdu: (56 commits) HDDS-15848. Add new upgrade action annotations for component versions (apache#10744) HDDS-15876. Remove ozone-csi (apache#10780) HDDS-14825. Add Grafana Dashboard and Metrics for ZDU (apache#10602) HDDS-15508. Add 2.2.0 to compatibility/upgrade test (apache#10782) HDDS-15867. [JDK25] Bump byte-buddy to 1.18.11 (apache#10776) HDDS-15809. Validate x-amz-copy-source-range on UploadPartCopy (apache#10711). HDDS-15678. OFS isDirectory/isFile should not trigger pipeline refresh or return block locations (apache#10741) HDDS-15171. Add available space check on follower during bootstrap. (apache#10185) HDDS-14043. Fix ls -e UnsupportedOperationException on ofs/o3fs (apache#10209) HDDS-15709. Replace random name suffix with uniqueObjectName (apache#10773) HDDS-15677. Update client side version checks to use `isSupportedBy`. (apache#10611) HDDS-15849. Force channel shutdown in XceiverClientGrpc.close (apache#10747) HDDS-15765. Fail fast when --node-id is omitted for OM compact/defrag on an HA cluster (apache#10699) HDDS-14989. Delay follower SCM DN server start until Ratis log catch-up. (apache#10617). HDDS-15456. Add SCM DB lookup and identify orphan(wrt SCM) and deleted-but-present containers (apache#10547) HDDS-15812. Rename test utils for naming convention (apache#10766) HDDS-15600. Fix ListObjects response for encoding-type and empty delimiter (apache#10586). HDDS-15656. Fix non-atomic data size accumulation in ContainerBalancer move callback (apache#10708) HDDS-15667. `containerbalancer status`: show stop reason and iteration details after stop (apache#10681) HDDS-15823. Fix flaky TestFailureHandlingByClient#testContainerExclusionWithClosedContainerException (apache#10721) ... Conflicts: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerPrepare.java

…ation * HDDS-14496-zdu: (57 commits) HDDS-14687. Add test that OM can finalize from a snapshot (apache#10832) HDDS-14580. Remove OM Prepare for Upgrade Code for ZDU (apache#9723) HDDS-15848. Add new upgrade action annotations for component versions (apache#10744) HDDS-15876. Remove ozone-csi (apache#10780) HDDS-14825. Add Grafana Dashboard and Metrics for ZDU (apache#10602) HDDS-15508. Add 2.2.0 to compatibility/upgrade test (apache#10782) HDDS-15867. [JDK25] Bump byte-buddy to 1.18.11 (apache#10776) HDDS-15809. Validate x-amz-copy-source-range on UploadPartCopy (apache#10711). HDDS-15678. OFS isDirectory/isFile should not trigger pipeline refresh or return block locations (apache#10741) HDDS-15171. Add available space check on follower during bootstrap. (apache#10185) HDDS-14043. Fix ls -e UnsupportedOperationException on ofs/o3fs (apache#10209) HDDS-15709. Replace random name suffix with uniqueObjectName (apache#10773) HDDS-15849. Force channel shutdown in XceiverClientGrpc.close (apache#10747) HDDS-15765. Fail fast when --node-id is omitted for OM compact/defrag on an HA cluster (apache#10699) HDDS-14989. Delay follower SCM DN server start until Ratis log catch-up. (apache#10617). HDDS-15456. Add SCM DB lookup and identify orphan(wrt SCM) and deleted-but-present containers (apache#10547) HDDS-15812. Rename test utils for naming convention (apache#10766) HDDS-15600. Fix ListObjects response for encoding-type and empty delimiter (apache#10586). HDDS-15656. Fix non-atomic data size accumulation in ContainerBalancer move callback (apache#10708) HDDS-15667. `containerbalancer status`: show stop reason and iteration details after stop (apache#10681) ... Conflicts: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMUpgradeFinalization.java hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto

…dation * HDDS-14496-zdu: (62 commits) HDDS-15875. OM should validate software versions of peers before accepting finalize command (apache#10777) HDDS-15417. Handle mixed datanode versions on the replication path (apache#10570) HDDS-15716. Replace layout version protos with new version-named protos in the DN–SCM protocol (apache#10817) HDDS-15715. Add helpful CLI flags to finalization CLIs (apache#10678) HDDS-14687. Add test that OM can finalize from a snapshot (apache#10832) HDDS-14580. Remove OM Prepare for Upgrade Code for ZDU (apache#9723) HDDS-15848. Add new upgrade action annotations for component versions (apache#10744) HDDS-15876. Remove ozone-csi (apache#10780) HDDS-14825. Add Grafana Dashboard and Metrics for ZDU (apache#10602) HDDS-15508. Add 2.2.0 to compatibility/upgrade test (apache#10782) HDDS-15867. [JDK25] Bump byte-buddy to 1.18.11 (apache#10776) HDDS-15809. Validate x-amz-copy-source-range on UploadPartCopy (apache#10711). HDDS-15678. OFS isDirectory/isFile should not trigger pipeline refresh or return block locations (apache#10741) HDDS-15171. Add available space check on follower during bootstrap. (apache#10185) HDDS-14043. Fix ls -e UnsupportedOperationException on ofs/o3fs (apache#10209) HDDS-15709. Replace random name suffix with uniqueObjectName (apache#10773) HDDS-15677. Update client side version checks to use `isSupportedBy`. (apache#10611) HDDS-15849. Force channel shutdown in XceiverClientGrpc.close (apache#10747) HDDS-15765. Fail fast when --node-id is omitted for OM compact/defrag on an HA cluster (apache#10699) HDDS-14989. Delay follower SCM DN server start until Ratis log catch-up. (apache#10617). ... Conflicts: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManager.java

Reduce rpc payload scope for isDirectory/isFile methods
f41cd8e

jojochuang self-requested a review July 13, 2026 16:19

jojochuang added the performance label Jul 13, 2026

adoroszlai added the AI-gen label Jul 13, 2026

jojochuang requested a review from Copilot July 14, 2026 22:48

Copilotstarted reviewing on behalf of jojochuang July 14, 2026 22:48 View session

CopilotAI reviewed Jul 14, 2026
View reviewed changes

jojochuang reviewed Jul 15, 2026
View reviewed changes

Comment thread...e/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestOFSIsDirectoryBenchmark.java

jojochuang reviewed Jul 15, 2026
View reviewed changes

Comment thread...zone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneFileSystem.java
Comment thread...zone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneFileSystem.java

addressed review comments
a33265a

yandrey321 requested a review from jojochuang July 15, 2026 21:51

jojochuang merged commit d2700a4 into apache:masterJul 16, 2026
45 checks passed

Navigation Menu

Uh oh!

HDDS-15678. OFS isDirectory/isFile should not trigger pipeline refresh or return block locations - #10741

HDDS-15678. OFS isDirectory/isFile should not trigger pipeline refresh or return block locations#10741
jojochuang merged 2 commits into
apache:masterfrom
yandrey321:HDDS-15678

yandrey321 commented Jul 13, 2026•
edited
Loading

Uh oh!

CopilotAI left a comment

Uh oh!

Uh oh!

jojochuang left a comment

Uh oh!

Uh oh!

jojochuangJul 15, 2026

Uh oh!

Uh oh!

jojochuang commented Jul 15, 2026

Uh oh!

jojochuang left a comment

Uh oh!

Uh oh!

Uh oh!

yandrey321 commented Jul 15, 2026

Uh oh!

jojochuang commented Jul 15, 2026

Uh oh!

Uh oh!

jojochuang commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

yandrey321 commented Jul 13, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jojochuangJul 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jojochuang commented Jul 15, 2026

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

yandrey321 commented Jul 15, 2026

Uh oh!

jojochuang commented Jul 15, 2026

Uh oh!

Uh oh!

jojochuang commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yandrey321 commented Jul 13, 2026•
edited
Loading