HDDS-15065. Replace Ratis Snapshot Trigger with DB Flush for Periodic Flush Operations by priyeshkaratha · Pull Request #10100 · apache/ozone

priyeshkaratha · 2026-04-21T06:45:03Z

What changes were proposed in this pull request?

This change addresses the SCM flush delay caused by the minimum transaction gap enforced in Apache Ratis.
Previously, even with the timer-based flush mechanism, DB updates were effectively delayed due to the default snapshot creation gap (~1024 transactions).

Changes:

Reduce ozone.scm.ha.ratis.server.snapshot.creation.gap from 1024 to 1
Remove this configuration from Ozone

Impact:

Reduces delay in SCM DB flush
Improves delete block processing latency
Simplifies configuration

What is the link to the Apache JIRA

HDDS-15065

How was this patch tested?

Tested using modified testcases.

ChenSammi · 2026-04-22T09:58:16Z

@sumitagrawl , for HDDS-8508(#4683), what was the consideration at that time, that triggers snapshot instead of calls transactionBuffer.flush() in SCMHATransactionBufferMonitorTask.run() ?

sumitagrawl · 2026-04-23T04:19:48Z

@sumitagrawl , for HDDS-8508(#4683), what was the consideration at that time, that triggers snapshot instead of calls transactionBuffer.flush() in SCMHATransactionBufferMonitorTask.run() ?

There is a discussion for setting up to '1' but after discussion with @szetszwo , its not practical scenario to have less than 1024 transaction to be flushed as it keeps having updated.

@szetszwo Please share your opinion

priyeshkaratha · 2026-04-23T19:48:32Z

@sumitagrawl @ChenSammi @szetszwo

The ozone.scm.ha.ratis.snapshot.threshold default of 1000 is smaller than the ozone.scm.ha.ratis.server.snapshot.creation.gap default of 1024. Because snapshot.creation.gap is the gate that Ratis applies to any manual snapshot API call (exactly the kind SCMHATransactionBufferMonitorTask makes), and because the auto-trigger always fires 24 entries earlier, the buffer monitor task never successfully triggers a snapshot under default configuration. It incurs an unnecessary Ratis RPC round-trip every 60 seconds that always goes nowhere, and any issue with the auto-trigger path has no time-based fallback as originally intended.

The fix is to either:

Raise ozone.scm.ha.ratis.snapshot.threshold to be strictly greater than ozone.scm.ha.ratis.server.snapshot.creation.gap (e.g., threshold = 2000, gap = 1024) so that the time-based monitor becomes the primary snapshot trigger for moderate transaction volumes, with auto-trigger as the backstop.
Lower ozone.scm.ha.ratis.server.snapshot.creation.gap below threshold (e.g., gap = 1, threshold = 1000) so the monitor can trigger aggressively in low-traffic situations without waiting for the auto-trigger.

ChenSammi · 2026-04-28T08:20:51Z

@priyeshkaratha , can you try call transactionBuffer.flush() in SCMHATransactionBufferMonitorTask.run()? per the offline discussion with Sumit.

szetszwo · 2026-04-28T18:01:00Z

There is a discussion for setting up to '1' ...

@sumitagrawl , What was the discussion? Could you remind me?

ChenSammi · 2026-04-30T07:11:37Z

@sumitagrawl , are we OK to go with this " transactionBuffer.flush()" approach? The CI is passed.

sumitagrawl

@priyeshkaratha Below things to be checked,

DB flush is done in monitor thread, earlier is synchronized via ratis call in takeSnapshot(). This may have some impact of partial flush.
Need recheck idompotency of all operation in SCM

ChenSammi · 2026-05-06T05:42:44Z

@priyeshkaratha Below things to be checked,

1. DB flush is done in monitor thread, earlier is synchronized via ratis call in takeSnapshot(). This may have some impact of partial flush.
2. Need recheck idompotency of all operation in SCM

@priyeshkaratha , please check and make sure that flush call is thread safe.
@sumitagrawl , "idompotency check " of all operation in SCM, we can do this in another thread, not related with this change. Flush is called multiple places in SCM, only takeSnapshot goes through ratis. SCM should have the capability to handle raft log reapply correctly.

I'm not very sure about the goal of SCMHATransactionBufferMonitorTask, but with the current default ozone.scm.ha.ratis.snapshot.threshold 1000, and ozone.scm.ha.ratis.server.snapshot.creation.gap 1024, SCMHATransactionBufferMonitorTask basically is a no-ops.

sumitagrawl · 2026-05-06T06:39:22Z

@szetszwo Shall we set ozone.scm.ha.ratis.server.snapshot.creation.gap to "1" ? when snapshot is taken via timer ?

I'm not very sure about the goal of SCMHATransactionBufferMonitorTask, but with the current default ozone.scm.ha.ratis.snapshot.threshold 1000, and ozone.scm.ha.ratis.server.snapshot.creation.gap 1024, SCMHATransactionBufferMonitorTask basically is a no-ops.

priyeshkaratha · 2026-05-08T05:46:41Z

@ChenSammi @sumitagrawl The current change with use of DB flush will impact of partial flush.
So fix will be better to make ozone.scm.ha.ratis.server.snapshot.creation.gap to 1.

Copilot

Pull request overview

This PR targets SCM HA DB flush latency by removing reliance on Apache Ratis snapshot triggering (which was previously impacted by a minimum snapshot-creation gap) and by eliminating the Ozone-side configuration knob for the snapshot creation gap.

Changes:

Remove ozone.scm.ha.ratis.server.snapshot.creation.gap from Ozone configuration (key + default.xml entry) and stop propagating it into Ratis properties.
Change SCM HA transaction buffer monitoring to flush the buffered DB operations directly rather than triggering a Ratis snapshot.
Update integration tests to stop setting the removed config key.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestStorageContainerManagerHA.java	Removes usage of the deleted snapshot-gap config in HA SCM integration test setup.
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestDatanodeSCMNodesReconfiguration.java	Removes usage of the deleted snapshot-gap config in reconfiguration test setup.
hadoop-ozone/integration-test-recon/src/test/java/org/apache/hadoop/ozone/recon/AbstractTestStorageDistributionEndpoint.java	Drops static import + config set for the removed snapshot-gap key.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMHATransactionBufferMonitorTask.java	Switches the periodic monitor from snapshot triggering to direct DB transaction-buffer flush.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMHAManagerImpl.java	Updates monitor task construction to no longer require `SCMRatisServer`.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/RatisUtil.java	Stops setting Ratis snapshot creation gap via Ozone config propagation.
hadoop-hdds/common/src/main/resources/ozone-default.xml	Removes the snapshot creation gap property from default configuration.
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java	Removes the snapshot creation gap config constant and default value.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

 private static void setRaftSnapshotProperties(
 final RaftProperties properties,
 final ConfigurationSource ozoneConf) {
 Snapshot.setAutoTriggerEnabled(properties, true);
 Snapshot.setAutoTriggerThreshold(properties,
 ozoneConf.getLong(ScmConfigKeys.OZONE_SCM_HA_RATIS_SNAPSHOT_THRESHOLD,
 ScmConfigKeys.OZONE_SCM_HA_RATIS_SNAPSHOT_THRESHOLD_DEFAULT));
- Snapshot.setCreationGap(properties,
- ozoneConf.getLong(ScmConfigKeys.OZONE_SCM_HA_RATIS_SNAPSHOT_GAP,
- ScmConfigKeys.OZONE_SCM_HA_RATIS_SNAPSHOT_GAP_DEFAULT));
 }


- // is reset to previous known value
- transactionBuffer.getLatestSnapshotRef().compareAndSet(
- null, lastSnapshot);
+ LOG.error("TransactionFlushTask is failed", e);


ChenSammi · 2026-05-11T03:23:15Z

The current change with use of DB flush will impact of partial flush.

What will cause partial flush?

sumitagrawl

LGTM

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

hadoop-hdds/common/src/main/resources/ozone-default.xml:4176

PR description says the snapshot creation gap is reduced from 1024 to 1, but the code change here removes the ozone.scm.ha.ratis.server.snapshot.creation.gap property entirely (and RatisUtil no longer sets it). Please update the PR description to match the implemented approach, or reintroduce a default/value of 1 if that was still intended.

 <property>
<name>ozone.scm.ha.dbtransactionbuffer.flush.interval</name>
<value>60s</value>
<tag>SCM, OZONE</tag>
<description>Wait duration for flush of buffered transaction.</description>
</property>

priyeshkaratha force-pushed the HDDS-15065 branch from aaff6af to e34f417Compare April 21, 2026 08:35

priyeshkaratha marked this pull request as ready for review April 23, 2026 20:19

priyeshkaratha added 2 commits April 29, 2026 14:17

HDDS-15065. Reduce Ratis snapshot gap to fix SCM flush delay.
d7781df

addressing review commits
9a8d822

priyeshkaratha force-pushed the HDDS-15065 branch from 0bc0dfe to 9a8d822Compare April 29, 2026 09:59

sumitagrawl reviewed May 6, 2026
View reviewed changes

ChenSammi requested a review from Copilot May 8, 2026 07:50

Copilotstarted reviewing on behalf of ChenSammi May 8, 2026 07:51 View session

CopilotAI reviewed May 8, 2026
View reviewed changes

priyeshkaratha requested a review from sumitagrawl May 26, 2026 08:34

priyeshkaratha force-pushed the HDDS-15065 branch from 6c9d7b8 to 181e5ffCompare May 26, 2026 09:09

priyeshkaratha changed the title ~~HDDS-15065. Reduce Ratis snapshot gap to fix SCM flush delay.~~HDDS-15065. Replace Ratis Snapshot Trigger with DB Flush for Periodic Flush OperationsMay 26, 2026

HDDS-15065. Handling partial flush scenarios
6948e64

priyeshkaratha force-pushed the HDDS-15065 branch from 181e5ff to 6948e64Compare May 27, 2026 07:10

sumitagrawl reviewed May 27, 2026
View reviewed changes

Comment threadhadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMStateMachine.java

addressing review comments
60a0ec0

sumitagrawl approved these changes May 29, 2026
View reviewed changes

ChenSammi requested a review from Copilot June 1, 2026 04:23

Copilotstarted reviewing on behalf of ChenSammi June 1, 2026 04:23 View session

CopilotAI reviewed Jun 1, 2026
View reviewed changes

sumitagrawl merged commit d2ae71e into apache:masterJun 1, 2026
47 checks passed

Uh oh!

Conversation

priyeshkaratha commented Apr 21, 2026

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

ChenSammi commented Apr 22, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sumitagrawl commented Apr 23, 2026

Uh oh!

priyeshkaratha commented Apr 23, 2026

Uh oh!

ChenSammi commented Apr 28, 2026

Uh oh!

szetszwo commented Apr 28, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ChenSammi commented Apr 30, 2026

Uh oh!

sumitagrawl left a comment

Choose a reason for hiding this comment

Uh oh!

ChenSammi commented May 6, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sumitagrawl commented May 6, 2026

Uh oh!

priyeshkaratha commented May 8, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

ChenSammi commented May 11, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

sumitagrawl left a comment

Choose a reason for hiding this comment

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ChenSammi commented Apr 22, 2026•
edited
Loading

szetszwo commented Apr 28, 2026•
edited
Loading

ChenSammi commented May 6, 2026•
edited
Loading

ChenSammi commented May 11, 2026•
edited
Loading