HDDS-13108. Refactor StorageVolume to use SlidingWindow by ptlrs · Pull Request #8843 · apache/ozone

ptlrs · 2025-07-22T06:37:23Z

Please describe your PR in detail:

This PR uses the new sliding window implementation.
It migrates all existing checks to detect a failed volume to use the new time-based sliding window utility.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13108

How was this patch tested?

CI:https://github.com/ptlrs/ozone/actions/runs/16436635030

… scanner failures

… tests

…e-failed-volume-checks-to-one-sliding-window

…g window mechanism

ptlrs · 2025-07-22T18:03:33Z

Hi @errose28 @Tejaskriya @adoroszlai can you please review this PR?

Tejaskriya

Thanks for working on this @ptlrs , please find a suggestion below

ptlrs · 2025-07-31T06:38:04Z

Thanks for the review @Tejaskriya. I have added the configuration.

Tejaskriya

Looks good to me, @errose28 could you please take a look?

github-actions · 2025-11-11T00:05:52Z

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

github-actions · 2025-11-25T00:05:10Z

Thank you for your contribution. This PR is being closed due to inactivity. If needed, feel free to reopen it.

ptlrs · 2026-02-09T19:29:33Z

Hi @errose28, could you please reopen this PR?

…e-failed-volume-checks-to-one-sliding-window # Conflicts: # hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeConfiguration.java # hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java

ptlrs · 2026-02-12T08:48:18Z

Hi @errose28, the conflicts have been resolved for this PR. Could you please take a look.

github-actions · 2026-03-06T00:13:27Z

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

…failed-volume-checks-to-one-sliding-window

adoroszlai

Thanks @ptlrs for the patch.

…k interval

…stomizable sliding window timestamps

Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

ptlrs · 2026-03-27T18:24:43Z

Hi @errose28 @ChenSammi @adoroszlai I have updated this PR, could you please take another look?

adoroszlai

Thanks @ptlrs for updating the patch, LGTM.

ChenSammi · 2026-03-30T06:39:11Z

 )
 private Duration diskCheckTimeout = DISK_CHECK_TIMEOUT_DEFAULT;

+ @Config(key = "hdds.datanode.disk.check.sliding.window.timeout",


With this sliding window introduced, "hdds.datanode.disk.check.io.test.count" property function is half removed. We should consider deprecate "hdds.datanode.disk.check.io.test.count" and introduce a new boolean property with name, like "hdds.datanode.disk.check.io.test.enabled".

To not break existing users, it's recommend to add new property "hdds.datanode.disk.check.io.test.enabled", instead of change the current property "hdds.datanode.disk.check.io.test.count" to "hdds.datanode.disk.check.io.test.enabled".

Ok, I reverted the change which removed the config and updated the deprecated config list.

…ust configurations for volume health checks

…ibility

…-one-sliding-window

ChenSammi · 2026-04-07T09:51:50Z

@ptlrs , it looks like we cannot add "hdds.datanode.disk.check.io.test.count" to DeprecationDelta. It's different property type.

ptlrs · 2026-04-07T18:24:36Z

I have removed the modifications to DeprecationDelta

ChenSammi · 2026-04-08T04:32:55Z

Thanks @ptlrs , and @Tejaskriya @adoroszlai for review.

ptlrs · 2026-04-09T17:52:19Z

Thanks for the reviews @ChenSammi @adoroszlai @Tejaskriya

ptlrs added 4 commits July 11, 2025 22:15

HDDS-13108. Transition to use the generic sliding window for counting…
b1df449
… scanner failures

HDDS-13108. Remove unused SlidingWindow implementation and associated…
625d52a
… tests

Merge remote-tracking branch 'upstream/master' into HDDS-13108-Migrat…
f108d3c
…e-failed-volume-checks-to-one-sliding-window

HDDS-13108. Replace count-based failure checks with time-based slidin…
33b9a4b
…g window mechanism

ptlrs marked this pull request as draft July 22, 2025 06:38

errose28 added the scanners Changes related to datanode container and volume scanners label Jul 22, 2025

Tejaskriya self-requested a review July 23, 2025 08:31

errose28 self-requested a review July 23, 2025 18:46

Tejaskriya reviewed Jul 29, 2025
View reviewed changes

Comment thread...ner-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/StorageVolume.java Outdated

HDDS-13108. Add configuration for sliding window timeout in disk checks
e81f59b

ptlrs requested a review from Tejaskriya July 31, 2025 06:38

Tejaskriya reviewed Aug 4, 2025
View reviewed changes

github-actionsBot added the stale label Nov 11, 2025

github-actionsBot closed this Nov 25, 2025

errose28 reopened this Feb 9, 2026

github-actionsBot removed the stale label Feb 10, 2026

HDDS-13108. Remove unused imports from HddsVolume
17fa938

github-actionsBot added the stale label Mar 6, 2026

Merge remote-tracking branch 'origin/master' into HDDS-13108-Migrate-…
4d8b6fe
…failed-volume-checks-to-one-sliding-window

adoroszlai reviewed Mar 9, 2026
View reviewed changes

Comment thread...tainer-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java Outdated
Comment thread...test/java/org/apache/hadoop/ozone/container/common/volume/TestStorageVolumeHealthChecks.java Outdated

adoroszlai removed the stale label Mar 9, 2026

adoroszlai changed the title ~~HDDS-13108. Migrate failed volume checks to one sliding window~~HDDS-13108. Refactor StorageVolume to use SlidingWindowMar 9, 2026

checkstyle
0b9e02b

ChenSammi reviewed Mar 11, 2026
View reviewed changes

Comment thread...c/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeConfiguration.java

ChenSammi reviewed Mar 11, 2026
View reviewed changes

Comment thread...ner-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/StorageVolume.java

ChenSammi reviewed Mar 11, 2026
View reviewed changes

Comment thread...ner-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/StorageVolume.java

ptlrs added 3 commits March 24, 2026 19:39

HDDS-13108. Enforce minimum sliding window timeout based on disk chec…
158819e
…k interval

HDDS-13108. Add Clock support to StorageVolume and its Builder for cu…
10d0dcd
…stomizable sliding window timestamps

HDDS-13108. Update sliding window checks and fix test clock adjustments
af0e4f8

adoroszlai reviewed Mar 25, 2026
View reviewed changes

Comment threadhadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/SlidingWindow.java Outdated

ptlrsand others added 2 commits March 25, 2026 08:30

Update SlidingWindow.java
1ec9e0e
Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

HDDS-13108. Fix Javadoc for MonotonicClock in SlidingWindow
0de347c

ptlrs requested review from ChenSammi and adoroszlai March 27, 2026 18:24

adoroszlai approved these changes Mar 27, 2026
View reviewed changes

Comment thread...test/java/org/apache/hadoop/ozone/container/common/volume/TestStorageVolumeHealthChecks.java

ptlrs mentioned this pull request Mar 29, 2026
HDDS-14871. DataNode: tolerate per-volume health-check latch timeouts before marking volumes failed. #9954
Merged

ChenSammi marked this pull request as ready for review March 30, 2026 04:25

ChenSammi reviewed Mar 30, 2026
View reviewed changes

Comment thread...c/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeConfiguration.java Outdated

ChenSammi reviewed Mar 30, 2026
View reviewed changes

HDDS-13108. Replace IO test count with disk check enable flag and adj…
9e20c2d
…ust configurations for volume health checks

ptlrs requested a review from ChenSammi March 31, 2026 03:13

HDDS-13108. Revert change to remove IO test count for backward compat…
b6d2478
…ibility

ChenSammi approved these changes Apr 7, 2026
View reviewed changes

Merge branch 'master' into HDDS-13108-Migrate-failed-volume-checks-to…
bdee24d
…-one-sliding-window

HDDS-13108. Remove deprecation delta
7629895

ChenSammi merged commit 9ff7eaa into apache:masterApr 8, 2026
45 checks passed

ptlrs deleted the HDDS-13108-Migrate-failed-volume-checks-to-one-sliding-window branch April 9, 2026 17:51

Uh oh!

Conversation

ptlrs commented Jul 22, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Please describe your PR in detail:

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

ptlrs commented Jul 22, 2025

Uh oh!

Tejaskriya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ptlrs commented Jul 31, 2025

Uh oh!

Tejaskriya left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Nov 11, 2025

Uh oh!

github-actionsBot commented Nov 25, 2025

Uh oh!

ptlrs commented Feb 9, 2026

Uh oh!

ptlrs commented Feb 12, 2026

Uh oh!

github-actionsBot commented Mar 6, 2026

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ptlrs commented Mar 27, 2026

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ChenSammiMar 30, 2026

Choose a reason for hiding this comment

Uh oh!

ptlrsMar 31, 2026

Choose a reason for hiding this comment

Uh oh!

ChenSammiApr 1, 2026

Choose a reason for hiding this comment

Uh oh!

ptlrsApr 3, 2026

Choose a reason for hiding this comment

Uh oh!

ChenSammi commented Apr 7, 2026

Uh oh!

ptlrs commented Apr 7, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ChenSammi commented Apr 8, 2026

Uh oh!

ptlrs commented Apr 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ptlrs commented Jul 22, 2025•
edited
Loading

ptlrs commented Apr 7, 2026•
edited
Loading