Skip to content

HDDS-13500. Transfer Non SST Files in the last batch of the tarball transfer. - #8857

Merged
sadanand48 merged 5 commits into
apache:masterfrom
sadanand48:HDDS-13500
Jul 30, 2025
Merged

HDDS-13500. Transfer Non SST Files in the last batch of the tarball transfer.#8857
sadanand48 merged 5 commits into
apache:masterfrom
sadanand48:HDDS-13500

Conversation

@sadanand48

@sadanand48sadanand48 commented Jul 24, 2025

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

In the current impl, all file types in the DB are transferred in every batch and subsequent requests exclude the already tranferred files. If the mod time for a file changes they would be resent in the next batch which means that a file like LOG would be transferred in every batch.

To avoid this we can send non-sst files only in the last batch and send in the .sst files in the earlier transfers.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13500

How was this patch tested?

Unit tests

@sadanand48sadanand48 added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label Jul 24, 2025
jojochuang
jojochuang reviewed Jul 25, 2025
View reviewed changes

@jojochuangjojochuang left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we want to add a few lines of explanation in the writeDBToArchive or writeDbDataToStream method, that as an optimization, onlySstFile is expected to be false only at the last batch iteration. Otherwise several months from now I won't remember a thing.

Comment thread...n-test/src/test/java/org/apache/hadoop/ozone/om/TestOMDbCheckpointServletInodeBasedXfer.java Outdated
jojochuang
jojochuang reviewed Jul 26, 2025
View reviewed changes

@jojochuangjojochuang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apart from these cosmetic issues this PR almost ready to go

Comment thread...n-test/src/test/java/org/apache/hadoop/ozone/om/TestOMDbCheckpointServletInodeBasedXfer.java Outdated
@sadanand48
sadanand48 marked this pull request as ready for review July 30, 2025 05:49
jonaswu2000

This comment was marked as duplicate.

Sign in to view

@sadanand48
sadanand48 merged commit 2870126 into apache:masterJul 30, 2025
90 of 93 checks passed
@sadanand48

sadanand48 commented Jul 30, 2025

Copy link
Copy Markdown
ContributorAuthor

Thanks @jojochuang for the review.

jojochuang pushed a commit to jojochuang/ozone that referenced this pull request Jul 31, 2025
@sadanand48@jojochuang
HDDS-13500. Transfer Non SST Files in the last batch of the tarball t…
23cc2f8
…ransfer. (apache#8857)
jojochuang added a commit to jojochuang/ozone that referenced this pull request Nov 19, 2025
@jojochuang
Revert "HDDS-13500. Transfer Non SST Files in the last batch of the t…
776b524
…arball transfer. (apache#8857)"
This reverts commit 2870126.
Conflicts:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java
Change-Id: Ib9141efa7e55b0f71520da8e16d32f5cd9b63331
@jojochuangjojochuang mentioned this pull request Nov 20, 2025
Closed
jojochuang added a commit that referenced this pull request Nov 21, 2025
@jojochuang
Revert "HDDS-13500. Transfer Non SST Files in the last batch of the t…
b2e8d01
…arball transfer. (#8857)"
This reverts commit 2870126.
Conflicts:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java
Change-Id: Ib9141efa7e55b0f71520da8e16d32f5cd9b63331
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jojochuangjojochuangjojochuang approved these changes
@swamirishiswamirishiAwaiting requested review from swamirishi
@smengclsmengclAwaiting requested review from smengcl
+1 more reviewer
@jonaswu2000jonaswu2000jonaswu2000 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

snapshothttps://issues.apache.org/jira/browse/HDDS-6517

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@sadanand48@jojochuang@jonaswu2000