HDDS-8389. [Snapshot] Added integration test for SnapDiff when OM leader failover happens by hemantk-12 · Pull Request #4657 · apache/ozone

hemantk-12 · 2023-05-04T20:11:16Z

What changes were proposed in this pull request?

Added integration test for SnapDiff when OM leader failover happens
Explicitly close snapshotDiffManager and invalidate snapshotCache upon OmSnapshotManager#close.
Shutdown snapDiffExecutor and sstDumpToolExecutor in SnapshotDiffManager#close.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-8389

How was this patch tested?

Ran newly added tests locally and verified they pass.

…der failover happens

smengcl · 2023-05-11T20:47:37Z

@swamirishi Could you take a look at this?

smengcl

Thanks @hemantk-12 . Great test additions. Nicely written.

swamirishi

Some comments

swamirishi · 2023-05-17T00:46:38Z

+
+ // If job was IN_PROGRESS or DONE state when OM restarted, it should be
+ // DONE by this time.
+ // If job FAILED during crash (which mostly happens in the test because


nit: Do we have a unit test case for the FAILED state?

No integration test for FAILED state. I added few tests as unit test in PR: https://github.com/apache/ozone/pull/4716/files#diff-11392a9810b911a6b87f9f4440cb5e5342b2a9c112851e7dceafa5103d999e1dR427

swamirishi · 2023-05-17T00:51:57Z

+ });
+ }
+
+ assertEquals(1, snapshotIds.stream().distinct().count());


Is snapshotId assertion enough? I guess we should also check rocks db contents.

Checking only snapshotId should be enough. Test is to make sure that snapshotId is consistent among all the OM nodes. We don't care about it value.
What do you mean by "check rocks db contents"?

the list of sst files should be same I believe.

I don't think that should be tested as part of this test.

smengcl · 2023-05-18T18:20:22Z

There are some conflicts in the test due to HDDS-8645.

hemantk-12 · 2023-05-22T22:11:52Z

There are some conflicts in the test due to HDDS-8645.

Resolved conflicts and also ran test 100 times to make sure there is no flakiness.

It passed all the time (Workflow failed because of timeout) : https://github.com/hemantk-12/ozone/actions/runs/5049594952/jobs/9059656428

smengcl

lgtm

smengcl · 2023-05-23T23:54:35Z

Thanks @hemantk-12 for the test addition, and @swamirishi for the review.

adoroszlai added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label May 4, 2023

hemantk-12 force-pushed the HDDS-8389 branch from 33910be to 77b7562Compare May 4, 2023 20:21

umamaheswararao requested a review from smengcl May 8, 2023 16:11

hemantk-12 mentioned this pull request May 8, 2023
HDDS-8530. [Snapshot] Fix for OM crash on restart due to snapshot chain manager corruption #4677
Merged

hemantk-12 marked this pull request as draft May 8, 2023 23:36

hemantk-12 force-pushed the HDDS-8389 branch 4 times, most recently from 1313363 to 116cd7cCompare May 10, 2023 02:05

HDDS-8389. [Snapshot] Added integration test for SnapDiff when OM lea…
480b998
…der failover happens

hemantk-12 force-pushed the HDDS-8389 branch from 116cd7c to 480b998Compare May 11, 2023 00:57

hemantk-12 marked this pull request as ready for review May 11, 2023 00:58

smengcl reviewed May 16, 2023
View reviewed changes

Comment threadhadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshot.java Outdated