Skip to content

HDDS-12745. Container checksum should be reported during container close and DN restart - #8204

Merged
aswinshakil merged 10 commits into
apache:HDDS-10239-container-reconciliationfrom
aswinshakil:HDDS-12745
Apr 15, 2025
Merged

HDDS-12745. Container checksum should be reported during container close and DN restart#8204
aswinshakil merged 10 commits into
apache:HDDS-10239-container-reconciliationfrom
aswinshakil:HDDS-12745

Conversation

@aswinshakil

@aswinshakilaswinshakil commented Mar 31, 2025

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

  1. The container checksum though generated during container close, is not being reported to the SCM.
  2. During DN restart we are not fetching the container checksum into KeyValueContainerData hence the SCM doesn't get the checksum.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-12745

How was this patch tested?

Integration Test

@aswinshakilaswinshakil self-assigned this Mar 31, 2025
@errose28errose28 mentioned this pull request Apr 3, 2025
Merged
errose28
errose28 reviewed Apr 4, 2025
View reviewed changes

@errose28errose28 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this @aswinshakil. Are there some existing container replication tests we can augment to test this as well?

Comment thread...e/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java Outdated
Comment thread...st/src/test/java/org/apache/hadoop/ozone/dn/checksum/TestContainerCommandReconciliation.java Outdated
Comment thread...st/src/test/java/org/apache/hadoop/ozone/dn/checksum/TestContainerCommandReconciliation.java
hemantk-12
hemantk-12 reviewed Apr 11, 2025
View reviewed changes

@hemantk-12hemantk-12 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @aswinshakil for the patch. Overall looks good to me apart from the existing comments.

Comment thread...st/src/test/java/org/apache/hadoop/ozone/dn/checksum/TestContainerCommandReconciliation.java Outdated
@aswinshakil

aswinshakil commented Apr 11, 2025

Copy link
Copy Markdown
MemberAuthor

Filed HDDS-12824 to optimize container checksum read during DN startup.

errose28
errose28 reviewed Apr 14, 2025
View reviewed changes

@errose28errose28 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few minor comments. Functionality LGTM when accounting for the follow up jira

Comment thread...e/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java
Comment thread...c/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerChecksumTreeManager.java
Comment thread...st/src/test/java/org/apache/hadoop/ozone/dn/checksum/TestContainerCommandReconciliation.java Outdated
errose28
errose28 approved these changes Apr 14, 2025
View reviewed changes

@errose28errose28 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix. LGTM.

@aswinshakil
aswinshakil merged commit 8768d04 into apache:HDDS-10239-container-reconciliationApr 15, 2025
errose28 added a commit to errose28/ozone that referenced this pull request May 6, 2025
@errose28
Merge branch 'HDDS-10239-container-reconciliation' into HDDS-10374-sc…
62d7794
…anner-builds-mt
* HDDS-10239-container-reconciliation:
HDDS-12849. On demand container scanner should not be static (apache#8296)
HDDS-12745. Container checksum should be reported during container close and DN restart (apache#8204)
Conflicts:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OnDemandContainerDataScanner.java
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerSet.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestOnDemandContainerDataScanner.java
@aswinshakilaswinshakil mentioned this pull request Jun 11, 2025
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hemantk-12hemantk-12hemantk-12 left review comments
@kerneltimekerneltimekerneltime approved these changes
@errose28errose28errose28 approved these changes

Assignees

@aswinshakilaswinshakil

Labels

container-reconciliation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@aswinshakil@kerneltime@hemantk-12@errose28