Skip to content

HDDS-15651. Test case for DiskBalancer when markContainerForDelete fails - #10593

Merged
adoroszlai merged 3 commits into
apache:masterfrom
arunsarin85:HDDS-15651
Jul 4, 2026
Merged

HDDS-15651. Test case for DiskBalancer when markContainerForDelete fails#10593
adoroszlai merged 3 commits into
apache:masterfrom
arunsarin85:HDDS-15651

Conversation

@arunsarin85

@arunsarin85arunsarin85 commented Jun 23, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Test-only PR for HDDS-15651. Adds two unit tests in TestDiskBalancerTask to document the intended DiskBalancer move/cleanup behavior when markContainerForDelete() fails or when lazy deletion fails.

Please describe your PR in detail:
DiskBalancer treats container move and source cleanup as separate phases. Once import and ContainerSet update succeed, the move is reported as success even if marking the old source replica fails. The old replica is queued in pendingDeletionContainers and removed after replica.deletion.delay.

This PR adds tests to lock in that behavior and document a known gap when lazy deletion fails.

Test 1: moveSucceedsWhenMarkContainerForDeleteFails

  • Simulates markContainerForDelete() failure on the source replica after a successful move.
  • Verifies the move is still reported as success (success metrics updated, no rollback).
  • Verifies ContainerSet points to the destination replica.
  • Verifies the source replica stays on disk temporarily and is queued for lazy deletion.
  • After the delay, verifies the source replica is removed via cleanupPendingDeletionContainers().

Test 2: lazyDeletionFailureDoesNotRetry

  • Runs a successful move and advances the clock past the deletion delay.
  • Mocks KeyValueContainerUtil.removeContainer() to fail during lazy deletion.
  • Verifies the source replica remains on disk, the pending queue entry is dropped, and deletion is not retried on a second cleanup attempt.
  • Documents current behavior when lazy deletion fails (recovery depends on other paths such as DN restart for Ratis).

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15651

How was this patch tested?

mvn test -pl hadoop-hdds/container-service -am
-Dtest=TestDiskBalancerTask#moveSucceedsWhenMarkContainerForDeleteFails,TestDiskBalancerTask#lazyDeletionFailureDoesNotRetry
-DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false

@arunsarin85
arunsarin85 marked this pull request as draft June 24, 2026 15:40
Gargi-jais11
Gargi-jais11 reviewed Jun 25, 2026
View reviewed changes

@Gargi-jais11Gargi-jais11 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @arunsarin85 for raisin the concern. I have left comments below to discuss on this.

Comment thread...ervice/src/main/java/org/apache/hadoop/ozone/container/diskbalancer/DiskBalancerService.java Outdated
Comment thread...ervice/src/main/java/org/apache/hadoop/ozone/container/diskbalancer/DiskBalancerService.java Outdated
@arunsarin85
arunsarin85 marked this pull request as ready for review June 25, 2026 08:10
@arunsarin85

arunsarin85 commented Jun 25, 2026

Copy link
Copy Markdown
ContributorAuthor

Thanks @Gargi-jais11 for the comments !
As per the design and feature flow explained in the jira https://issues.apache.org/jira/browse/HDDS-15651
I have modified this PR to be a test only [added 2 additional tests.] . Will modify the description accordingly.

adoroszlai
adoroszlai reviewed Jun 25, 2026
View reviewed changes
Comment thread...rvice/src/test/java/org/apache/hadoop/ozone/container/diskbalancer/TestDiskBalancerTask.java Outdated
Comment thread...rvice/src/test/java/org/apache/hadoop/ozone/container/diskbalancer/TestDiskBalancerTask.java Outdated
@adoroszlaiadoroszlai changed the title HDDS-15651. Roll back DiskBalancer move when markContainerForDelete failsHDDS-15651. Test case for DiskBalancer when markContainerForDelete failsJun 25, 2026
ayushtkn
ayushtkn approved these changes Jul 1, 2026
View reviewed changes

@ayushtknayushtkn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor nitpicks rest lgtm

Comment thread...rvice/src/test/java/org/apache/hadoop/ozone/container/diskbalancer/TestDiskBalancerTask.java Outdated
Comment thread...rvice/src/test/java/org/apache/hadoop/ozone/container/diskbalancer/TestDiskBalancerTask.java Outdated
Gargi-jais11
Gargi-jais11 reviewed Jul 2, 2026
View reviewed changes

@Gargi-jais11Gargi-jais11 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM! Let's fix few of the review comments above.

@arunsarin85

arunsarin85 commented Jul 3, 2026•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Overall LGTM! Let's fix few of the review comments above.

Fixed

ayushtkn
ayushtkn approved these changes Jul 4, 2026
View reviewed changes

@ayushtknayushtkn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@adoroszlai
adoroszlai merged commit b97a11b into apache:masterJul 4, 2026
30 checks passed
@adoroszlai

adoroszlai commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Thanks @arunsarin85 for the patch, @ayushtkn, @Gargi-jais11 for the review.

@arunsarin85
arunsarin85 deleted the HDDS-15651 branch July 5, 2026 05:59
errose28 added a commit to errose28/ozone that referenced this pull request Jul 7, 2026
@errose28
Merge branch 'master' into HDDS-14908-rescan-container-if-updated
04bca00
* master: (519 commits)
HDDS-14544. OM DB Insights: Duplicate API calls triggered when changing limit selector (apache#10677).
HDDS-15587. [Recon] Show 0 for offline DN pending deletion instead of -1 (apache#10585).
HDDS-15521. StreamBlockInputStream fails with TimeoutIOException without retry or failover. (apache#10479)
HDDS-15170. Add mock-based unit tests for DataStream write path (apache#10230)
HDDS-15552. Ratis events should not be published as metrics (apache#10523)
HDDS-15746. Bump kerby to 2.1.2 (apache#10666)
HDDS-15579. Replace SimpleSpanProcessor with BatchSpanProcessor (apache#10569)
HDDS-15747. Address review comments for HDDS-15083 (apache#10669)
HDDS-15732. Some ozone commands ignore OZONE_MODULE_ACCESS_ARGS (apache#10655)
HDDS-15605. Fix flaky testContainerExclusionWithClosedContainerException (apache#10621)
HDDS-11855. Fix flaky TestContainerBalancerDatanodeNodeLimit#checkIterationResultException (apache#10667)
HDDS-15741. Bump awssdk to 2.46.17 (apache#10661)
HDDS-10307. Speed up TestOzoneManagerHAWithStoppedNodes (apache#10658)
HDDS-15651. Test case for DiskBalancer when markContainerForDelete fails (apache#10593)
HDDS-15742. Bump nimbus-jose-jwt to 10.9.1 (apache#10662)
HDDS-15719. Add check for allowed action usage in workflows (apache#10641)
HDDS-15744. Bump javassist to 3.32.0-GA (apache#10665)
HDDS-15737. Fix intermittent failure in balancerShouldOnlySelectConfiguredIncludeContainers (apache#10660)
HDDS-15743. Bump gson to 2.14.0 (apache#10664)
HDDS-11093. Fix intermittent failure in TestContainerBalancerDatanodeNodeLimit#testMetrics (apache#10659)
...
Conflicts:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerScanHelper.java
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adoroszlaiadoroszlaiadoroszlai left review comments
@Gargi-jais11Gargi-jais11Gargi-jais11 left review comments
@ayushtknayushtknayushtkn approved these changes
@ChenSammiChenSammiAwaiting requested review from ChenSammi

Assignees

No one assigned

Labels

test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@arunsarin85@adoroszlai@ayushtkn@Gargi-jais11