Skip to content

HDDS-12397. Persist putBlock for closed container. - #7943

Merged
kerneltime merged 8 commits into
apache:HDDS-10239-container-reconciliationfrom
aswinshakil:HDDS-12397
Mar 6, 2025
Merged

HDDS-12397. Persist putBlock for closed container.#7943
kerneltime merged 8 commits into
apache:HDDS-10239-container-reconciliationfrom
aswinshakil:HDDS-12397

Conversation

@aswinshakil

@aswinshakilaswinshakil commented Feb 21, 2025•
edited
Loading

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

The putBlock call for OPEN container cannot be used for the CLOSED container and needs some modification to update bscId and remove changes that are only applicable to OPEN container such a pending put block cache, hsync support.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-12397

How was this patch tested?

Added Unit Test.

@aswinshakilaswinshakil self-assigned this Feb 21, 2025
kerneltime
kerneltime reviewed Feb 25, 2025
View reviewed changes
Comment thread...vice/src/test/java/org/apache/hadoop/ozone/container/keyvalue/impl/TestBlockManagerImpl.java Outdated
Comment thread...vice/src/test/java/org/apache/hadoop/ozone/container/keyvalue/impl/TestBlockManagerImpl.java Outdated
errose28
errose28 reviewed Feb 25, 2025
View reviewed changes
Comment thread...-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/BlockManagerImpl.java
}

@Override
public long putBlockForClosedContainer(Container container, BlockData data, boolean overwriteBcsId)

@errose28errose28Feb 25, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need a flag to overwrite the BCSID? Shouldn't it always be an automatic greater-than check?

Comment thread...-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/BlockManagerImpl.java Outdated
Comment thread...-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/BlockManagerImpl.java Outdated
Comment thread...-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/BlockManagerImpl.java Outdated
hemantk-12
hemantk-12 reviewed Feb 27, 2025
View reviewed changes
Comment thread...-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/BlockManagerImpl.java Outdated
Comment thread...-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/BlockManagerImpl.java
}

@Override
public long putBlockForClosedContainer(Container container, BlockData data, boolean overwriteBcsId)

@hemantk-12hemantk-12Feb 26, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: there are lots of warnings due to raw use Container. Maybe we change BlockManager to type T something like this BlockManager<T extends ContainerData>. It could be a separate PR if we agree.

@aswinshakilaswinshakilMar 3, 2025

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. We can do that in the main branch as an improvement.

Comment thread...-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/BlockManagerImpl.java Outdated
Comment thread...vice/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/FilePerBlockStrategy.java Outdated
errose28
errose28 reviewed Mar 3, 2025
View reviewed changes

@errose28errose28 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates @aswinshakil

Comment thread...vice/src/test/java/org/apache/hadoop/ozone/container/keyvalue/impl/TestBlockManagerImpl.java Outdated
Comment thread...vice/src/test/java/org/apache/hadoop/ozone/container/keyvalue/impl/TestBlockManagerImpl.java Outdated
Comment thread...vice/src/test/java/org/apache/hadoop/ozone/container/keyvalue/impl/TestBlockManagerImpl.java
Comment thread...vice/src/test/java/org/apache/hadoop/ozone/container/keyvalue/impl/TestBlockManagerImpl.java
Comment thread...vice/src/test/java/org/apache/hadoop/ozone/container/keyvalue/impl/TestBlockManagerImpl.java
Comment thread.../src/test/java/org/apache/hadoop/ozone/container/keyvalue/impl/TestFilePerBlockStrategy.java
Comment thread...vice/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/FilePerBlockStrategy.java Outdated
Comment thread...-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/BlockManagerImpl.java Outdated
Comment thread...-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/BlockManagerImpl.java Outdated
Comment thread...-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/BlockManagerImpl.java
errose28
errose28 reviewed Mar 6, 2025
View reviewed changes
Comment thread...vice/src/test/java/org/apache/hadoop/ozone/container/keyvalue/impl/TestBlockManagerImpl.java Outdated
Comment thread...vice/src/test/java/org/apache/hadoop/ozone/container/keyvalue/impl/TestBlockManagerImpl.java Outdated
Comment thread...vice/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/FilePerBlockStrategy.java Outdated
errose28
errose28 approved these changes Mar 6, 2025
View reviewed changes

@errose28errose28 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the continued work on this @aswinshakil

@kerneltime
kerneltime merged commit be53ed7 into apache:HDDS-10239-container-reconciliationMar 6, 2025
@aswinshakilaswinshakil mentioned this pull request Mar 10, 2025
Merged
errose28 added a commit to errose28/ozone that referenced this pull request Apr 11, 2025
@errose28
Merge branch 'HDDS-10239-container-reconciliation' into HDDS-10374-sc…
15d6848
…anner-builds-mt
* HDDS-10239-container-reconciliation: (646 commits)
HDDS-11763. Implement container repair logic within datanodes. (apache#7474)
HDDS-11887. Recon - Identify container replicas difference based on content checksums (apache#7942)
HDDS-12397. Persist putBlock for closed container. (apache#7943)
HDDS-12361. Mark testGetSnapshotDiffReportJob as flaky
HDDS-12375. Random object created and used only once (apache#7933)
HDDS-12335. Fix ozone admin namespace summary to give complete output (apache#7908)
HDDS-12364. Require Override annotation for overridden methods (apache#7923)
HDDS-11530. Support listMultipartUploads max uploads and markers (apache#7817)
HDDS-11867. Remove code paths for non-Ratis SCM. (apache#7911)
HDDS-12363. Add import options to IntelliJ IDEA style settings (apache#7921)
HDDS-12215. Mark testContainerStateMachineRestartWithDNChangePipeline as flaky
HDDS-12331. BlockOutputStream.failedServers is not thread-safe (apache#7885)
HDDS-12188. Move server-only upgrade classes from hdds-common to hdds-server-framework (apache#7903)
HDDS-12362. Remove temporary checkstyle suppression file (apache#7920)
HDDS-12343. Fix spotbugs warnings in Recon (apache#7902)
HDDS-12286. Fix license headers and imports for ozone-tools (apache#7919)
HDDS-12284. Fix license headers and imports for ozone-s3-secret-store (apache#7917)
HDDS-12275. Fix license headers and imports for ozone-integration-test (apache#7904)
HDDS-12164. Rename and deprecate DFSConfigKeysLegacy config keys (apache#7803)
HDDS-12283. Fix license headers and imports for ozone-recon-codegen (apache#7916)
...
Conflicts:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerMerkleTreeWriter.java
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerCheck.java
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/ContainerMerkleTreeTestUtils.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerMerkleTreeWriter.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainerCheck.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestBackgroundContainerDataScanner.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestOnDemandContainerDataScanner.java
hadoop-ozone/dist/src/main/smoketest/admincli/container.robot
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/dn/scanner/TestBackgroundContainerDataScannerIntegration.java
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/dn/scanner/TestBackgroundContainerMetadataScannerIntegration.java
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/dn/scanner/TestOnDemandContainerDataScannerIntegration.java
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hemantk-12hemantk-12hemantk-12 left review comments
@kerneltimekerneltimekerneltime approved these changes
@errose28errose28errose28 approved these changes

Assignees

@aswinshakilaswinshakil

Labels

container-reconciliation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@aswinshakil@kerneltime@hemantk-12@errose28