HDDS-12984. Use InodeID to identify the SST files inside the tarball. by sadanand48 · Pull Request #8477 · apache/ozone

sadanand48 · 2025-05-19T08:32:43Z

What changes were proposed in this pull request?

Use InodeID to identify the SST files inside the tarball : Add Inode named files inside the tarball

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-12984

How was this patch tested?

Unit test

swamirishi

Thanks for the patch @sadanand48 I have taken an overall look at the code changes I see we are trying to force fit to the previous algo which doesn't necessarily have to be the case here. I have left comments inline for the changes required.

swamirishi · 2025-05-19T11:08:21Z

Let us copy the code for this implementation, since the algorithm might change significantly and in order to keep the implementation clean it is ok to have some intersection.

swamirishi

thanks @sadanand48 for addressing the previous review comments left few more review comments inline

jojochuang

ideally public method should all have test coverage.

jojochuang

This is looking good to me.

peterxcli · 2025-06-25T16:46:02Z

Is the new class OMDBCheckpointServletInodeBasedXfer used in production code? Cause I dont see any reference of it?

sadanand48 · 2025-06-25T16:58:55Z

@peterxcli not as of now, I will add it in follow up PR's.

* master: (90 commits) HDDS-12468. Check for space availability for all dns while container creation in pipeline (apache#8663) HDDS-12151. Fail write when volume is full considering min free space (apache#8642) HDDS-13251. Update Byteman usage README license (apache#8700) HDDS-13251. Support dynamic Byteman scripts via bmsubmit in ozonesecure-ha (apache#8654) HDDS-12070. Bump Ratis to 3.2.0 (apache#8689) HDDS-12984. Use InodeID to identify the SST files inside the tarball. (apache#8477) HDDS-13319. Simplify KeyPrefixFilter (apache#8692) HDDS-13295. Remove jackson1 exclusions for hadoop-common (apache#8687) HDDS-13314. Remove unused maven-pdf-plugin (apache#8686) HDDS-13324. Optimize memory footprint for Recon listKeys API (apache#8680) HDDS-13240. Add newly added metrics into grafana dashboard. (apache#8656) HDDS-13309. Add keyIterator/valueIterator methods to Table. (apache#8675) HDDS-13325. Introduce OZONE_SERVER_OPTS for common options for server processes (apache#8685) HDDS-13318. Simplify the getRangeKVs methods in Table (apache#8683) HDDS-13322. Remove module auto-detection from flaky-test-check (apache#8679) HDDS-13289. Remove usage of Jetty StringUtil (apache#8684) HDDS-13270. Reduce getBucket API invocations in S3 bucket owner verification (apache#8653) HDDS-13288. Container checksum file proto changes to account for deleted blocks. (apache#8649) HDDS-13306. Intermittent failure in testDirectoryDeletingServiceIntervalReconfiguration (apache#8682) HDDS-13317. Table should support empty array/String (apache#8676) ...

* master: (170 commits) HDDS-12468. Check for space availability for all dns while container creation in pipeline (apache#8663) HDDS-12151. Fail write when volume is full considering min free space (apache#8642) HDDS-13251. Update Byteman usage README license (apache#8700) HDDS-13251. Support dynamic Byteman scripts via bmsubmit in ozonesecure-ha (apache#8654) HDDS-12070. Bump Ratis to 3.2.0 (apache#8689) HDDS-12984. Use InodeID to identify the SST files inside the tarball. (apache#8477) HDDS-13319. Simplify KeyPrefixFilter (apache#8692) HDDS-13295. Remove jackson1 exclusions for hadoop-common (apache#8687) HDDS-13314. Remove unused maven-pdf-plugin (apache#8686) HDDS-13324. Optimize memory footprint for Recon listKeys API (apache#8680) HDDS-13240. Add newly added metrics into grafana dashboard. (apache#8656) HDDS-13309. Add keyIterator/valueIterator methods to Table. (apache#8675) HDDS-13325. Introduce OZONE_SERVER_OPTS for common options for server processes (apache#8685) HDDS-13318. Simplify the getRangeKVs methods in Table (apache#8683) HDDS-13322. Remove module auto-detection from flaky-test-check (apache#8679) HDDS-13289. Remove usage of Jetty StringUtil (apache#8684) HDDS-13270. Reduce getBucket API invocations in S3 bucket owner verification (apache#8653) HDDS-13288. Container checksum file proto changes to account for deleted blocks. (apache#8649) HDDS-13306. Intermittent failure in testDirectoryDeletingServiceIntervalReconfiguration (apache#8682) HDDS-13317. Table should support empty array/String (apache#8676) ... Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerChecksumTreeManager.java

…tarball. (apache#8477)" This reverts commit 96390ac. Conflicts: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMDbCheckpointServletInodeBasedXfer.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java Change-Id: I777c0c8145830df85098afde7b89ff7864472534

…tarball. (#8477)" This reverts commit 96390ac. Conflicts: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMDbCheckpointServletInodeBasedXfer.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java Change-Id: I777c0c8145830df85098afde7b89ff7864472534

swamirishi requested changes May 19, 2025
View reviewed changes

jojochuang added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label May 19, 2025

jojochuang reviewed May 20, 2025
View reviewed changes

Comment thread...ne-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckPointServletInodeBasedXfer.java Outdated

SaketaChalamchala reviewed May 28, 2025
View reviewed changes

Comment thread...ne-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckPointServletInodeBasedXfer.java Outdated

SaketaChalamchala reviewed May 28, 2025
View reviewed changes

Comment thread...ne-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckPointServletInodeBasedXfer.java Outdated

SaketaChalamchala reviewed May 28, 2025
View reviewed changes

Comment thread...ne-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckPointServletInodeBasedXfer.java Outdated

SaketaChalamchala reviewed May 28, 2025
View reviewed changes

Comment thread...ne-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckPointServletInodeBasedXfer.java Outdated

jojochuang requested review from SaketaChalamchala, jojochuang and swamirishi June 5, 2025 15:07

swamirishi requested changes Jun 6, 2025
View reviewed changes

swamirishi reviewed Jun 6, 2025
View reviewed changes

Comment thread...ne-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java Outdated

swamirishi reviewed Jun 6, 2025
View reviewed changes

Comment thread...ne-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java Outdated

swamirishi reviewed Jun 6, 2025
View reviewed changes

Comment thread...ne-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java

swamirishi requested changes Jun 7, 2025
View reviewed changes

Comment thread...ne-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java Outdated

swamirishi mentioned this pull request Jun 8, 2025
HDDS-12134. Implement Snapshot Cache lock for OM Bootstrap #8474
Merged

jojochuang requested a review from swamirishi June 10, 2025 22:25

swamirishi requested changes Jun 11, 2025
View reviewed changes

swamirishi reviewed Jun 11, 2025
View reviewed changes

Comment thread...ne-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java

swamirishi reviewed Jun 11, 2025
View reviewed changes

Comment thread...ne-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java Outdated

swamirishi reviewed Jun 12, 2025
View reviewed changes

Comment thread...ne-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java
Comment thread...ne-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java Outdated

jojochuang reviewed Jun 13, 2025
View reviewed changes

Comment thread...ne-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java Outdated

jojochuang reviewed Jun 13, 2025
View reviewed changes

Comment threadhadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/Archiver.java Outdated

jojochuang reviewed Jun 13, 2025
View reviewed changes

SaketaChalamchala reviewed Jun 13, 2025
View reviewed changes

Comment threadhadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServlet.java Outdated

SaketaChalamchala reviewed Jun 13, 2025
View reviewed changes

Comment thread...ne-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java

SaketaChalamchala reviewed Jun 13, 2025
View reviewed changes

Comment thread...ne-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java

SaketaChalamchala reviewed Jun 13, 2025
View reviewed changes

Comment thread...ne-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java

Refactor DB Checkpoint Utilities
5da6899

Use InodeID to identify the SST files inside the tarball
d87b2e9

sadanand48 force-pushed the HDDS-12984 branch from 5662aa0 to d87b2e9Compare June 16, 2025 12:45

Sadanand Shenoy added 3 commits June 16, 2025 20:53

findbugs fix
a25e594

Merge branch 'master' into HDDS-12984
ab7ab44

refactor
561c206

sadanand48 marked this pull request as ready for review June 18, 2025 18:05

jojochuang reviewed Jun 19, 2025
View reviewed changes

Comment thread...p-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotUtils.java Outdated
Comment threadhadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/Archiver.java

jojochuang reviewed Jun 20, 2025
View reviewed changes

Comment threadhadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/Archiver.java Outdated
Comment threadhadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/Archiver.java

jojochuang reviewed Jun 20, 2025
View reviewed changes

Sadanand Shenoy added 7 commits June 20, 2025 10:43

change format of hardlink file to suit existing impl
44e67c5

address comments
1e1f989

Tests for Archiver method
73d534e

findbugs fix
e995f56

Merge branch 'master' into HDDS-12984
5f7abb7

fix tests
5588e69

Merge branch 'master' into HDDS-12984
0665665

jojochuang requested a review from swamirishi June 23, 2025 22:58

jojochuang approved these changes Jun 23, 2025
View reviewed changes

jojochuang merged commit 96390ac into apache:masterJun 25, 2025

jojochuang mentioned this pull request Nov 20, 2025
HDDS-13908. Disable OMDBCheckpointServletInodeBasedXfer in Ozone 2.1 #9333
Closed

Uh oh!

Conversation

sadanand48 commented May 19, 2025

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

swamirishi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

swamirishi commented May 19, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

swamirishi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

peterxcli commented Jun 25, 2025

Uh oh!

sadanand48 commented Jun 25, 2025

Uh oh!

Reviewers