Skip to content

HDDS-7974. [Snapshot] KeyDeletingService to be aware of Ozone snapshots - #4486

Merged
aswinshakil merged 7 commits into
apache:masterfrom
aswinshakil:HDDS-7974
Apr 4, 2023
Merged

HDDS-7974. [Snapshot] KeyDeletingService to be aware of Ozone snapshots#4486
aswinshakil merged 7 commits into
apache:masterfrom
aswinshakil:HDDS-7974

Conversation

@aswinshakil

@aswinshakilaswinshakil commented Mar 28, 2023•
edited
Loading

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Refactored KeyDeletingService to accommodate deleting keys from snapshot DB's deletedTable.

  • Add support for snapshots in OMKeyPurgeRequest and OMKeyPurgeResponse
  • Abstracted common code for both SnapshotDeletingService and KeyDeletingService into AbstractKeyDeletingService
  • Added getBlocksForKeyDelete() which gets the BlockGroup for a single key from OmKeyLocationInfoGroup

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-7974

How was this patch tested?

Unit test and Existing test.

@aswinshakilaswinshakil added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label Mar 28, 2023
@aswinshakilaswinshakil self-assigned this Mar 28, 2023
smengcl
smengcl approved these changes Mar 28, 2023
View reviewed changes

@smengclsmengcl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @aswinshakil . I figure I would post the first part of my review while I continue reviewing the KDT refactoring.

Comment threadhadoop-ozone/interface-storage/src/main/java/org/apache/hadoop/ozone/om/OMMetadataManager.java
Comment threadhadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
Comment threadhadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshot.java
Comment thread...ne/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyPurgeRequest.java Outdated
Comment thread...ne/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyPurgeRequest.java
Comment thread.../ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyPurgeResponse.java Outdated
Comment threadhadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto Outdated
smengcl
smengcl requested changes Mar 28, 2023
View reviewed changes
Comment thread...one-manager/src/main/java/org/apache/hadoop/ozone/om/service/AbstractKeyDeletingService.java Outdated
Comment thread...one-manager/src/main/java/org/apache/hadoop/ozone/om/service/AbstractKeyDeletingService.java Outdated
Comment thread...one-manager/src/main/java/org/apache/hadoop/ozone/om/service/AbstractKeyDeletingService.java Outdated
Comment thread...one-manager/src/main/java/org/apache/hadoop/ozone/om/service/AbstractKeyDeletingService.java Outdated
Comment on lines +81 to +84
// TODO: Once HA and non-HA paths are merged, we should have
// only one code path here. Purge keys should go through an
// OMRequest model.
delCount = deleteAllKeys(results, manager);

@smengclsmengclMar 28, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not even sure OM can still boot without Ratis anymore. So this code path might never be used.

Need to double check.

Comment thread...one-manager/src/main/java/org/apache/hadoop/ozone/om/service/AbstractKeyDeletingService.java Outdated
Comment thread...one-manager/src/main/java/org/apache/hadoop/ozone/om/service/AbstractKeyDeletingService.java Outdated
Comment thread...ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/KeyDeletingService.java Outdated
Comment thread.../ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/SnapshotDeletingService.java
Comment thread.../ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/SnapshotDeletingService.java
Comment thread.../ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/SnapshotDeletingService.java Outdated
smengcl
smengcl reviewed Mar 30, 2023
View reviewed changes
Comment threadhadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto Outdated
smengcl
smengcl reviewed Mar 30, 2023
View reviewed changes
Comment thread...ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/KeyDeletingService.java
@@ -130,7 +129,7 @@ public BackgroundTaskResult call() throws Exception {
.getPendingDeletionKeys(keyLimitPerTask);

@smengclsmengclMar 30, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can add a separate timer inside getPendingDeletionKeys() later. Just a thought.

Comment threadhadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java Outdated
smengcl
smengcl approved these changes Mar 31, 2023
View reviewed changes

@smengclsmengcl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall lgtm

Comment threadhadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java Outdated
Comment threadhadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java Outdated
Comment threadhadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java Outdated
Comment threadhadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
Comment threadhadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
@aswinshakil

aswinshakil commented Apr 4, 2023

Copy link
Copy Markdown
MemberAuthor

Thanks @smengcl for the review.

@aswinshakil
aswinshakil merged commit 3adae01 into apache:masterApr 4, 2023
errose28 added a commit to errose28/ozone that referenced this pull request Apr 6, 2023
@errose28
Merge branch 'master' into HDDS-8115
6207a08
* master: (155 commits)
update readme (apache#4535)
HDDS-8374. Disable flaky unit test: TestContainerStateCounts
HDDS-8016. updated the ozone doc for linked bucket and deletion async limitation (apache#4526)
HDDS-8237. [Snapshot] loadDb() used by SstFiltering service creates extraneous directories. (apache#4446)
HDDS-8035. Intermittent timeout in TestOzoneManagerHAWithData.testOMHAMetrics (apache#4362)
HDDS-8039. Allow container inspector to run from ozone debug. (apache#4337)
HDDS-8304. [Snapshot] Reduce flakiness in testSkipTrackingWithZeroSnapshot (apache#4487)
HDDS-7974. [Snapshot] KeyDeletingService to be aware of Ozone snapshots (apache#4486)
HDDS-8368. ReplicationManager: Create ContainerReplicaOp with correct target Datanode (apache#4532)
HDDS-8358. Fix the space usage comparator in ContainerBalancerSelectionCriteria (apache#4527)
HDDS-8359. ReplicationManager: Fix getContainerReplicationHealth() so that it builds ContainerCheckRequest correctly (apache#4528)
HDDS-8361. Useless object in TestOzoneBlockTokenIdentifier (apache#4517)
HDDS-8325. Consolidate and refine RocksDB metrics of services (apache#4506)
HDDS-8135. Incorrect synchronization during certificate renewal in DefaultCertificateClient. (apache#4381)
HDDS-8127. Exclude deleted containers from Recon container count (apache#4440)
HDDS-8364. ReadReplicas may give wrong results with topology-aware read enabled (apache#4522)
HDDS-8354. Avoid WARNING about ObjectEndpoint#get (apache#4515)
HDDS-8324. DN data cache gets removed randomly asking for data from disk (apache#4499)
HDDS-8291. Upgrade to Hadoop 3.3.5 (apache#4484)
HDDS-8355. Mark TestOMRatisSnapshots#testInstallSnapshot as flaky
...
@smengclsmengcl mentioned this pull request Apr 14, 2023
Merged
2 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@smengclsmengclsmengcl approved these changes
@prashantpogdeprashantpogdeAwaiting requested review from prashantpogde

Assignees

@aswinshakilaswinshakil

Labels

snapshothttps://issues.apache.org/jira/browse/HDDS-6517

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@aswinshakil@smengcl