HDDS-15208. OM should learn to finalize from SCM after finalization command by sodonnel · Pull Request #10236 · apache/ozone

sodonnel · 2026-05-11T11:54:08Z

What changes were proposed in this pull request?

To finalize a cluster, first a command will be sent to OM. OM will make a call to SCM to trigger finalization on the HDDS layer and then save a finalization_in_progress key into the OM database indicating finalization has been started. This is to persist it incase of restarts.

When OM is started, it checks if finalization is needed. If so, it spawns a background service which polls to see if it is time to finalize or not.

On each poll, it will first check if the finalization_in_progress key is there. If so, it will poll SCM to see if the HDDS layer finalization is completed and only trigger OM finalization if SCM has completed.

If all criteria is met, the OM finalization call will be made via Ratis and then the background service will shutdown.

The poll interval defaults to 60 seconds, but its configurable via ozone.om.upgrade.finalization.check.interval.

In this PR, all the tests mock the OM finalization command, a followup will add the new command and remove the mocked part from the tests.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15208

How was this patch tested?

Existing integration tests were modified for the new flow. New unit tests have been added for the new polling service.

errose28

Thanks for adding this. Overall looks good, just left some minor comments inline.

…kground

…appen

This reverts commit 9afb239.

This reverts commit 4376bdf.

This reverts commit 324cf4a.

This reverts commit 93ac731.

errose28

Thanks for the updates. My understanding is that this is being done in two PRs, where this one adds the OM server side handling of finalization and a follow-up PR will add the client command that starts the background service and writes the finalizing key. Can you update the PR description?

Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/DatanodeStorage.java

errose28

Just two comments left. Looks like there's some new test failures to address as well.

sodonnel · 2026-06-03T16:58:16Z

@errose28 If the CI goes green I think all comments have been addressed. Please have a look when you have time.

* HDDS-14496-zdu: HDDS-15483. More renames missed in HDDS-15375 (apache#10436) HDDS-15376. Implement new ScmVersionManager. (apache#10367) HDDS-15208. OM should learn to finalize from SCM after finalization command (apache#10236) HDDS-15375. Renames to prepare to switch SCM to the new versioning framework. (apache#10366) Conflicts: hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/TestHDDSVersion.java hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/TestOzoneManagerVersion.java hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/upgrade/UpgradeUtils.java hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/upgrade/TestHDDSLayoutFeature.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DatanodeInfo.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManager.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java

* HDDS-14496-zdu: HDDS-15377. Switch SCM to use new ScmVersionManager (apache#10368) HDDS-15483. More renames missed in HDDS-15375 (apache#10436) HDDS-15376. Implement new ScmVersionManager. (apache#10367) HDDS-15208. OM should learn to finalize from SCM after finalization command (apache#10236) HDDS-15375. Renames to prepare to switch SCM to the new versioning framework. (apache#10366) Conflicts: hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/TestHDDSVersion.java hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/TestOzoneManagerVersion.java hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/upgrade/TestHDDSLayoutFeature.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DatanodeInfo.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/upgrade/FinalizationManagerImpl.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/upgrade/FinalizationStateManager.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/upgrade/FinalizationStateManagerImpl.java hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java

* HDDS-14496-zdu: HDDS-15379. Remove unused classes from the old upgrade framework (apache#10370) HDDS-15377. Switch SCM to use new ScmVersionManager (apache#10368) HDDS-15483. More renames missed in HDDS-15375 (apache#10436) HDDS-15376. Implement new ScmVersionManager. (apache#10367) HDDS-15208. OM should learn to finalize from SCM after finalization command (apache#10236) HDDS-15375. Renames to prepare to switch SCM to the new versioning framework. (apache#10366) Conflicts: hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/TestHDDSVersion.java hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/TestOzoneManagerVersion.java hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/DatanodeStorage.java hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/upgrade/HDDSLayoutFeature.java hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/upgrade/TestHDDSLayoutFeature.java hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestHSyncUpgrade.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/multitenant/TestMultiTenantVolume.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOmSnapshot.java hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/upgrade/package-info.java

* HDDS-14496-zdu: HDDS-15386. Remove LayoutFeature interface (apache#10376) HDDS-15379. Remove unused classes from the old upgrade framework (apache#10370) HDDS-15377. Switch SCM to use new ScmVersionManager (apache#10368) HDDS-15483. More renames missed in HDDS-15375 (apache#10436) HDDS-15376. Implement new ScmVersionManager. (apache#10367) HDDS-15208. OM should learn to finalize from SCM after finalization command (apache#10236) HDDS-15375. Renames to prepare to switch SCM to the new versioning framework. (apache#10366) Conflicts: hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/TestHDDSVersion.java hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/TestOzoneManagerVersion.java hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/upgrade/DatanodeVersionManager.java hadoop-hdds/framework/src/main/java/org/apache/hadoop/ozone/upgrade/RatisBasedVersionManager.java hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/upgrade/TestHDDSLayoutFeature.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMStorageConfig.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/upgrade/ScmVersionManager.java hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/upgrade/TestScmVersionManager.java hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestHSyncUpgrade.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/TestDNDataDistributionFinalization.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/TestScmDataDistributionFinalization.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/TestScmHAFinalization.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/multitenant/TestMultiTenantVolume.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/service/TestBlockDeletionService.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOmSnapshot.java hadoop-ozone/mini-cluster/src/main/java/org/apache/hadoop/ozone/UniformDatanodesFactory.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/upgrade/OMVersionManager.java hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/upgrade/package-info.java

* HDDS-14496-zdu: HDDS-15484. Decouple ComponentVersionManager from Storage (apache#10437) HDDS-15386. Remove LayoutFeature interface (apache#10376) HDDS-15379. Remove unused classes from the old upgrade framework (apache#10370) HDDS-15377. Switch SCM to use new ScmVersionManager (apache#10368) HDDS-15483. More renames missed in HDDS-15375 (apache#10436) HDDS-15376. Implement new ScmVersionManager. (apache#10367) HDDS-15208. OM should learn to finalize from SCM after finalization command (apache#10236) HDDS-15375. Renames to prepare to switch SCM to the new versioning framework. (apache#10366) Conflicts: hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/AbstractComponentVersionTest.java hadoop-hdds/docs/content/design/upgrade-dev-primer.md hadoop-hdds/framework/src/main/java/org/apache/hadoop/ozone/upgrade/ComponentVersionManager.java hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/upgrade/TestHDDSLayoutFeature.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMStorageConfig.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/upgrade/ScmVersionManager.java hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/upgrade/TestScmVersionManager.java hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestHSyncUpgrade.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/TestDNDataDistributionFinalization.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/TestScmDataDistributionFinalization.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/TestScmHAFinalization.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/multitenant/TestMultiTenantVolume.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/service/TestBlockDeletionService.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOmSnapshot.java hadoop-ozone/mini-cluster/src/main/java/org/apache/hadoop/ozone/UniformDatanodesFactory.java hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/upgrade/package-info.java hadoop-ozone/recon/pom.xml hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/upgrade/ReconUpgradeActionProvider.java

* HDDS-14496-zdu: HDDS-15374. Switch Recon to the new versioning framework (apache#10443) HDDS-15484. Decouple ComponentVersionManager from Storage (apache#10437) HDDS-15386. Remove LayoutFeature interface (apache#10376) HDDS-15379. Remove unused classes from the old upgrade framework (apache#10370) HDDS-15377. Switch SCM to use new ScmVersionManager (apache#10368) HDDS-15483. More renames missed in HDDS-15375 (apache#10436) HDDS-15376. Implement new ScmVersionManager. (apache#10367) HDDS-15208. OM should learn to finalize from SCM after finalization command (apache#10236) HDDS-15375. Renames to prepare to switch SCM to the new versioning framework. (apache#10366)

* HDDS-14496-zdu: HDDS-15374. Switch Recon to the new versioning framework (apache#10443) HDDS-15484. Decouple ComponentVersionManager from Storage (apache#10437) HDDS-15386. Remove LayoutFeature interface (apache#10376) HDDS-15379. Remove unused classes from the old upgrade framework (apache#10370) HDDS-15377. Switch SCM to use new ScmVersionManager (apache#10368) HDDS-15483. More renames missed in HDDS-15375 (apache#10436) HDDS-15376. Implement new ScmVersionManager. (apache#10367) HDDS-15208. OM should learn to finalize from SCM after finalization command (apache#10236) Conflicts: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/upgrade/InitialConstraintUpgradeAction.java hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/upgrade/ReconUpgradeAction.java hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/upgrade/UnhealthyContainerReplicaMismatchAction.java hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/upgrade/UnhealthyContainersStateContainerIdIndexUpgradeAction.java

* HDDS-14496-zdu: (363 commits) HDDS-15639. Adjust Upgrade status command to call OM intead of SCM (apache#10579) HDDS-15622. New finalize command should check OM server version (apache#10548) HDDS-15609. Legacy SCM Finalize command should become a no-op (apache#10543) HDDS-15528. Adjust upgrade finalize command to call OM instread of SCM (apache#10493) HDDS-15488. Recon upgrade actions should be idempotent (apache#10442) HDDS-15482. Add fencing based on datanode versions to SCM and Recon (apache#10504) HDDS-15374. Switch Recon to the new versioning framework (apache#10443) HDDS-15484. Decouple ComponentVersionManager from Storage (apache#10437) HDDS-15386. Remove LayoutFeature interface (apache#10376) HDDS-15379. Remove unused classes from the old upgrade framework (apache#10370) HDDS-15377. Switch SCM to use new ScmVersionManager (apache#10368) HDDS-15483. More renames missed in HDDS-15375 (apache#10436) HDDS-15376. Implement new ScmVersionManager. (apache#10367) HDDS-15208. OM should learn to finalize from SCM after finalization command (apache#10236) HDDS-15375. Renames to prepare to switch SCM to the new versioning framework. (apache#10366) HDDS-15195. Switch Datanode to the new versioning framework (apache#10215) HDDS-15263. Downgrade should happen in reverse order (apache#10269) HDDS-14821. Version specific data validation in rolling-upgrade suite (apache#10033) HDDS-15196. Split HDDS upgrade actions into specific SCM and Datanode actions. (apache#10206) HDDS-14670. SCM queryFinalizeStatus shouldFinalize should only be true when out of safemode (apache#10193) ... Conflicts: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMUpgradeFinalization.java hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/utils/OzoneManagerRatisUtils.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/upgrade/OMCancelPrepareRequest.java hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerStateMachine.java hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/protocolPB/TestOzoneManagerRequestHandler.java

S O'Donnell added 8 commits May 8, 2026 11:47

HDDS-15208. OM should learn to finalize from SCM
06d15e3

Remove log messages
6d85457

Fix findbugs
136574b

Fix checkstyle
8847638

Fix config check and update unit tests to verify ratis interactions
6581218

Fix style
2626446

Stop background service thread after finalization no longer needed
5823be4

Fix typo in ozone-default
cf7bbb2

github-actionsBot added the zdu Pull requests for Zero Downtime Upgrade (ZDU) https://issues.apache.org/jira/browse/HDDS-14496 label May 11, 2026

errose28 self-requested a review May 11, 2026 15:44

errose28 reviewed May 11, 2026
View reviewed changes

S O'Donnell added 19 commits May 12, 2026 11:26

Address review comments
77fbc35

Fix failing test
e042fad

Finalization should goto system audit log as its happening in the bac…
7dad7f1
…kground

Fix compile issue
ce82754

Add a meta-table key that must be present for finalization of OM to h…
8e6ddbe
…appen

Fix checkstyle
118498b

Fix test failures and meta table setting / deleting
7e3d80f

Increase wait timeout for finalization
9afb239

Debug messages
4376bdf

More debug
324cf4a

Merge branch 'HDDS-14496-zdu' into HDDS-15208
a910da5

Fix issue where DN was defaulting to incorrect layout version
93ac731

Revert "Increase wait timeout for finalization"
58b0a26
This reverts commit 9afb239.

Revert "Debug messages"
993fedc
This reverts commit 4376bdf.

Revert "More debug"
5f03e6b
This reverts commit 324cf4a.

Fix failing finalize service tests
b6a07c7

Fix checkstyle
b02c116

Fix failing container upgrade test with Ethan's patch
9352c04

Revert "Fix issue where DN was defaulting to incorrect layout version"
7282b36
This reverts commit 93ac731.

Fix DN default versions
0e6a85f

errose28 reviewed May 28, 2026
View reviewed changes

S O'Donnell added 7 commits May 28, 2026 11:39

Revert unintentional log format change
287a696

Also delete system audit log on test finish
1dba392

Use ratis transaction as epoch
66fd9f5

Use same random clientID
fd950a7

Always increment run count and add to logs
f9f0be1

Fix log4j settings
c8325a7

Merge branch 'HDDS-14496-zdu' into HDDS-15208
8d0bcd5
Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/DatanodeStorage.java

errose28 reviewed May 28, 2026
View reviewed changes

S O'Donnell added 2 commits May 29, 2026 17:48

Fix bad merge affecting DN version
2fa62f2

Fixes to auditlog.properties
9d39168

sodonnel changed the title ~~HDDS-15208. OM should learn to finalize from SCM~~HDDS-15208. OM should learn to finalize from SCM after finalization commandJun 2, 2026

S O'Donnell added 2 commits June 3, 2026 15:27

Add tests for finalize key manipulation in request / response handling
1f3e6b2

Fix find bugs
4eaa1a8

errose28 approved these changes Jun 5, 2026
View reviewed changes

errose28 merged commit eba78b6 into apache:HDDS-14496-zduJun 5, 2026
89 of 91 checks passed

Uh oh!

Conversation

sodonnel commented May 11, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

errose28 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

errose28 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

errose28 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sodonnel commented Jun 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sodonnel commented May 11, 2026•
edited
Loading