Skip to content

HDDS-13780. Skeleton of background snapshot defrag service - #9133

Merged
smengcl merged 34 commits into
apache:masterfrom
smengcl:HDDS-13780-skeleton-defrag-service
Oct 14, 2025
Merged

HDDS-13780. Skeleton of background snapshot defrag service#9133
smengcl merged 34 commits into
apache:masterfrom
smengcl:HDDS-13780-skeleton-defrag-service

Conversation

@smengcl

@smengclsmengcl commented Oct 9, 2025

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This is #9117 but shrunk down to just the skeleton of the service, without actual defrag operations.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13780

How was this patch tested?

  • Test case was running fine before the last commit that removes core defrag logic.

@smengcl
HDDS-13009. Background snapshot defrag service
8423ac7
@smengcl
Make RDBSstFileWriter public to be used in defrag service.
838cd43
(cherry picked from commit f81e5b7)
@smengcl
Implement delete(key) in RDBSstFileWriter.
61b9ffc
(cherry picked from commit e15a62d)
@smengcl
Add config ozone.snapshot.defrag.service.timeout, `ozone.snapshot.d…
73625a8
…efrag.limit.per.task`, `ozone.snapshot.defrag.service.interval`.
(cherry picked from commit c04d67a)
@smengcl
Add comments/TODOs for defrag service interval/timeout
f442502
@smengcl
Implement SnapshotDefragService
8e37e5d
(cherry picked from commit ac58394)
@smengcl
Make it compile
08cde3d
@smengcl
Checkstyle
2cd2cbf
@smengcl
Add a test case for debugging
6a8adb5
(cherry picked from commit ab9d67e)
@smengcl
Handle yaml previousSnapshotId null
0a131dd
@smengcl
Use snapshot chain iterator in findFirstSnapshotNeedingDefrag
668799d
@smengcl
Merge remote-tracking branch 'asf' into HDDS-13009-defrag-service
93db705
Conflicts:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalDataYaml.java
@smengcl
Handle yaml snapId null so that it won't cause NPE. Null check should…
135100e
… be handled in upper logic
@smengcl
Remove findPreviousDefraggedSnapshot
f5fd06a
@smengcl
Remove TODO that is not applicable
e970a3d
@smengcl
Add new configs to ozone-default.xml
c3de774
@smengcl
Clean up test case
92256eb
@smengcl
Logging and TODO cleanups.
9d57b89
@smengcl
Remove POC defrag logic and test case, keeping the skeleton
4a3e01a
@smengclsmengcl added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label Oct 9, 2025
CopilotAI reviewed Oct 9, 2025
View reviewed changes

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements a skeleton for a background snapshot defragmentation service in Ozone Manager. The service is designed to process snapshots in the active snapshot chain and defragment them by creating compacted RocksDB instances.

  • Adds the main SnapshotDefragService class with background task scheduling and snapshot processing logic
  • Integrates the service into KeyManager with configuration and lifecycle management
  • Updates related classes to support the defragmentation infrastructure

Reviewed Changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file
FileDescription
SnapshotDefragService.javaNew background service implementing snapshot defragmentation with full and incremental processing modes
OmSnapshotLocalDataYaml.javaFixed null handling for previous snapshot ID in YAML data construction
KeyManagerImpl.javaAdded service startup/shutdown and configuration integration for snapshot defragmentation
KeyManager.javaAdded interface method for accessing snapshot defrag service
OMConfigKeys.javaAdded configuration keys for defrag service interval and limits
RocksDatabase.javaMade checkpoint creation and get methods public for external access
RDBSstFileWriter.javaMade class and constructor public, added delete method for SST file operations
ozone-default.xmlAdded configuration properties for snapshot defragmentation service
OzoneConsts.javaAdded constant for defragmented checkpoint directory
OzoneConfigKeys.javaAdded timeout configuration for snapshot defragmentation service

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment threadhadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java Outdated
Comment threadhadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java Outdated
Comment threadhadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java
Comment threadhadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
swamirishi
swamirishi requested changes Oct 13, 2025
View reviewed changes

@swamirishiswamirishi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@smengcl thank you for working on the patch have a few review comments

Comment threadhadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java Outdated
Comment threadhadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBSstFileWriter.java
Comment threadhadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java Outdated
Comment threadhadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java Outdated
Comment threadhadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java Outdated
Comment threadhadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java
Comment threadhadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java Outdated
@smengcl

smengcl commented Oct 13, 2025

Copy link
Copy Markdown
ContributorAuthor

@smengcl thank you for working on the patch have a few review comments

Thanks @swamirishi for the review. I have addressed all comments. Pls take another look.

swamirishi
swamirishi approved these changes Oct 14, 2025
View reviewed changes

@swamirishiswamirishi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM @smengcl thank you for addressing the review comments

Comment threadhadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java Outdated
@smengcl
Revert package-private
eee47f3
@smengcl
Merge remote-tracking branch 'asf' into HDDS-13780-skeleton-defrag-se…
48c47e4
…rvice
Conflicts:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalDataYaml.java
@smengcl
Use new API getOmSnapshotLocalData(SnapshotInfo) that comes with th…
f8e3077
…e refactoring
@smengcl
smengcl merged commit 96ce0b6 into apache:masterOct 14, 2025
43 checks passed
@smengcl
smengcl deleted the HDDS-13780-skeleton-defrag-service branch October 14, 2025 20:06
@smengcl

smengcl commented Oct 14, 2025

Copy link
Copy Markdown
ContributorAuthor

Thanks @swamirishi , @TaiJuWu for reviewing this.

This was referenced Oct 31, 2025
Closed
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@swamirishiswamirishiswamirishi approved these changes
Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
+1 more reviewer
@TaiJuWuTaiJuWuTaiJuWu left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

snapshothttps://issues.apache.org/jira/browse/HDDS-6517

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@smengcl@TaiJuWu@swamirishi