HDDS-10926. Block deletion should update container merkle tree. by errose28 · Pull Request #6875 · apache/ozone

errose28 · 2024-06-27T17:57:06Z

What changes were proposed in this pull request?

When a block is deleted from a container, it should be marked as deleted in the container merkle tree. This tells peer datanodes that the data be skipped if they have not yet processed the deletion but find there is an error with this block. The chunk, block, and container hashes in the tree should not change.

This test also moves some existing test methods to a test util class since more tests, including those in this patch, need to use them.

What is the link to the Apache JIRA

HDDS-10926

How was this patch tested?

Unit tests added

* HDDS-10239-container-reconciliation: HDDS-10372. SCM and Datanode communication for reconciliation (apache#6506) HDDS-10239. Storage Container Reconciliation. (apache#6121)

* HDDS-10887-mt-manager: Rename data tree to container tree

kerneltime · 2024-06-28T20:34:39Z

Will give another look post addressing of review comments and rebase.

…-delete * HDDS-10239-container-reconciliation: HDDS-10887. Implement a basic Merkle Tree Manager. (apache#6778) HDDS-10923. Container Scanner should still scan unhealthy containers. (apache#6809) Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerChecksumTreeManager.java

…-delete * HDDS-10239-container-reconciliation: (184 commits) HDDS-10373. Implement framework for capturing Merkle Tree Metrics. (apache#6864) HDDS-11188. Initial setup for new UI layout and enable users to switch to new UI (apache#6953) HDDS-11120. Rich rebalancing status info (apache#6911) HDDS-11187. Fix Event Handling in Recon OMDBUpdatesHandler to Prevent ClassCastException. (apache#6950) HDDS-11213. Bump commons-daemon to 1.4.0 (apache#6971) HDDS-11212. Bump commons-net to 3.11.1 (apache#6973) HDDS-11211. Bump assertj-core to 3.26.3 (apache#6972) HDDS-11210. Bump log4j2 to 2.23.1 (apache#6970) HDDS-11150. Recon Overview page crashes due to failed API Calls (apache#6944) HDDS-11183. Keys from DeletedTable and DeletedDirTable of AOS should be deleted on batch operation while creating a snapshot (apache#6946) HDDS-11198. Fix Typescript configs for Recon (apache#6961) HDDS-11180. Simplify HttpServer2#inferMimeType return statement (apache#6963) HDDS-11194. OM missing audit log for upgrade (apache#6958) HDDS-10389. Implement a search feature for users to locate open keys within the Open Keys Insights section. (apache#6231) HDDS-10561. Dashboard for delete key metrics (apache#6948) HDDS-11192. Increase SPNEGO URL test coverage (apache#6956) HDDS-11179. DBConfigFromFile#readFromFile result of toIOException not thrown (apache#6957) HDDS-11186. First container log missing from bundle (apache#6952) HDDS-10844. Clarify snapshot create error message. (apache#6955) HDDS-11166. Switch to Rocky Linux-based ozone-runner (apache#6942) ...

This will be handled properly once HDDS-11254 is complete.

errose28 · 2024-07-31T17:06:03Z

For tests to pass I had to add the same metrics unregister workaround from #6945. ~~HDDS-10239~~HDDS-11254 will properly close the checksum tree manager when everything is connected and this will no longer be necessary.

errose28 · 2024-07-31T19:25:57Z

Run on my fork is green, waiting for verification here as well.

aswinshakil

Thanks for the patch @errose28. LGTM+1

smengcl · 2024-08-06T22:17:20Z

Thanks @errose28 . Nice commit messages btw :D

…p-supervisor Merge conflicts are resolved but the change does not yet build. * HDDS-10239-container-reconciliation: (183 commits) HDDS-10376. Add a Datanode API to supply a merkle tree for a given container. (apache#6945) HDDS-11289. Bump docker-maven-plugin to 0.45.0 (apache#7024) HDDS-11287. Code cleanup in XceiverClientSpi (apache#7043) HDDS-11283. Refactor KeyValueStreamDataChannel to avoid spurious IDE build issues (apache#7040) HDDS-11257. Ozone write does not work when http proxy is set for the JVM. (apache#7036) HDDS-11249. Bump ozone-runner to 20240729-jdk17-1 (apache#7003) HDDS-10517. Recon - Add a UI component for showing DN decommissioning detailed status and info (apache#6724) HDDS-10926. Block deletion should update container merkle tree. (apache#6875) HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test. (apache#7021) HDDS-11272. Statistics some node status information (apache#7025) HDDS-11278. Move code out of Hadoop util package (apache#7028) HDDS-11274. (addendum) Replace Hadoop annotations/configs with Ozone-specific ones HDDS-11274. Replace Hadoop annotations/configs with Ozone-specific ones (apache#7026) HDDS-11280. Add Synchronize in AbstractCommitWatcher.addAckDataLength (apache#7032) HDDS-11235. Spare InfoBucket RPC call in FileSystem#mkdir() call. (apache#6990) HDDS-11273. Bump commons-compress to 1.26.2 (apache#7023) HDDS-11225. Increase ipc.server.read.threadpool.size (apache#7007) HDDS-11224. Increase hdds.datanode.handler.count (apache#7011) HDDS-11259. [hsync] DataNode should verify HBASE_SUPPORT layout version for every PutBlock. (apache#7012) HDDS-11214. Added config to set rocksDB's max log file size and num of log files (apache#7014) ... Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java

…rrupt-files * HDDS-10239-container-reconciliation: (183 commits) HDDS-10376. Add a Datanode API to supply a merkle tree for a given container. (apache#6945) HDDS-11289. Bump docker-maven-plugin to 0.45.0 (apache#7024) HDDS-11287. Code cleanup in XceiverClientSpi (apache#7043) HDDS-11283. Refactor KeyValueStreamDataChannel to avoid spurious IDE build issues (apache#7040) HDDS-11257. Ozone write does not work when http proxy is set for the JVM. (apache#7036) HDDS-11249. Bump ozone-runner to 20240729-jdk17-1 (apache#7003) HDDS-10517. Recon - Add a UI component for showing DN decommissioning detailed status and info (apache#6724) HDDS-10926. Block deletion should update container merkle tree. (apache#6875) HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test. (apache#7021) HDDS-11272. Statistics some node status information (apache#7025) HDDS-11278. Move code out of Hadoop util package (apache#7028) HDDS-11274. (addendum) Replace Hadoop annotations/configs with Ozone-specific ones HDDS-11274. Replace Hadoop annotations/configs with Ozone-specific ones (apache#7026) HDDS-11280. Add Synchronize in AbstractCommitWatcher.addAckDataLength (apache#7032) HDDS-11235. Spare InfoBucket RPC call in FileSystem#mkdir() call. (apache#6990) HDDS-11273. Bump commons-compress to 1.26.2 (apache#7023) HDDS-11225. Increase ipc.server.read.threadpool.size (apache#7007) HDDS-11224. Increase hdds.datanode.handler.count (apache#7011) HDDS-11259. [hsync] DataNode should verify HBASE_SUPPORT layout version for every PutBlock. (apache#7012) HDDS-11214. Added config to set rocksDB's max log file size and num of log files (apache#7014) ... Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerChecksumTreeManager.java

errose28 added 19 commits June 5, 2024 21:48

Initial outline of checksum manager and merkle tree with proto
bf0dd6e

Merge branch 'HDDS-10239-container-reconciliation' into mt-manager-wip
408cff9
* HDDS-10239-container-reconciliation: HDDS-10372. SCM and Datanode communication for reconciliation (apache#6506) HDDS-10239. Storage Container Reconciliation. (apache#6121)

Add config key for lock stripes
3adbe5a

Keep deleted block list sorted
e02a609

Change chunk checksum proto, add the first passing test
3680a2e

Use config defaults for test setup
0d5800c

Finish tests for ContainerMerkleTree
eb01da4

Add tests for checksum maanger and standardize proto names
f35effd

Updates after reviewing diff
21dd870

Rename checksum manager and file. Fix findbugs and Rat
47cd213

Make checksum file read/write for the API based on files, not protos
6460fe7

Initially mark where blocks will be added
ff55f27

Block deleting task supports updating the file
fffeba6

Basic block delete test passes
4822941

Add test for block delete commands retried
6087fa3

Rename data tree to container tree
9714259

Merge branch 'HDDS-10887-mt-manager' into HDDS-10926-mt-delete
aacac83
* HDDS-10887-mt-manager: Rename data tree to container tree

Use generic container type in checksum manager
3161ad8

Checkstyle
041c5c2

github-actionsBot added the container-reconciliation label Jun 27, 2024

kerneltime reviewed Jun 28, 2024
View reviewed changes

Comment thread...service/src/test/java/org/apache/hadoop/ozone/container/common/TestBlockDeletingService.java Outdated

kerneltime reviewed Jun 28, 2024
View reviewed changes

Comment thread...service/src/test/java/org/apache/hadoop/ozone/container/common/TestBlockDeletingService.java

kerneltime reviewed Jun 28, 2024
View reviewed changes

Comment thread...c/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerChecksumTreeManager.java Outdated

errose28 commented Jul 23, 2024
View reviewed changes

Comment thread...tainer-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java

errose28 mentioned this pull request Jul 26, 2024
HDDS-10376. Add a Datanode API to supply a merkle tree for a given container. #6945
Merged

errose28 added 3 commits July 26, 2024 13:40

Use block name in task
245844c

Update test description
51e7020

errose28 added 5 commits July 30, 2024 15:32

Move test helper methods to util class
1748205

Add more tests for blocks being written to the file
5d24704

Checkstyle
0145809

Fix rat
52207ce

Add metrics unregister workaround from WIP HDDS-10376
a82c863
This will be handled properly once HDDS-11254 is complete.

errose28 marked this pull request as ready for review July 31, 2024 19:25

kerneltime approved these changes Aug 5, 2024
View reviewed changes

aswinshakil approved these changes Aug 6, 2024
View reviewed changes

errose28 merged commit d7f302e into apache:HDDS-10239-container-reconciliationAug 6, 2024

Navigation Menu

Uh oh!

HDDS-10926. Block deletion should update container merkle tree. - #6875

HDDS-10926. Block deletion should update container merkle tree.#6875
errose28 merged 28 commits into
apache:HDDS-10239-container-reconciliationfrom
errose28:HDDS-10926-mt-delete

errose28 commented Jun 27, 2024•
edited
Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kerneltime commented Jun 28, 2024

Uh oh!

Uh oh!

errose28 commented Jul 31, 2024•
edited
Loading

Uh oh!

errose28 commented Jul 31, 2024

Uh oh!

aswinshakil left a comment

Uh oh!

smengcl commented Aug 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

errose28 commented Jun 27, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kerneltime commented Jun 28, 2024

Uh oh!

Uh oh!

errose28 commented Jul 31, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

errose28 commented Jul 31, 2024

Uh oh!

aswinshakil left a comment

Choose a reason for hiding this comment

Uh oh!

smengcl commented Aug 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

errose28 commented Jun 27, 2024•
edited
Loading

errose28 commented Jul 31, 2024•
edited
Loading