HDDS-5051. Ensure failover to suggested leader if any for NotLeaderException. by bshashikant · Pull Request #2141 · apache/ozone

bshashikant · 2021-04-09T13:51:29Z

What changes were proposed in this pull request?

Handling of suggested leader for SCM client requests

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5051

How was this patch tested?

Added UT

bharatviswa504

Still reviewing.

bharatviswa504

Also, I think if we can have more mock tests for check with RetriableExceptionsWithNoFailOver and NonRetriableException and normal exceptions that would be good. This will give better test coverage and uncover any misses.

bshashikant · 2021-04-21T12:30:46Z

https://issues.apache.org/jira/browse/HDDS-5130 is open to track the test coverage issue.

…ception.

bharatviswa504

Overall LGTM.
One minor comment.

bharatviswa504

+1 (Pending CI)

…ing-upgrade-master-merge2 * upstream/master: (56 commits) HDDS-2212. Genconf tool should generate config files for secure clust… (apache#1788) HDDS-5166. Remove duplicate assignment of OZONE_OPTS for freon and sh (apache#2195) Revert "HDDS-5144. Create github check to alert when dependency tree is changed (apache#2177)" HDDS-4983. Display key offset for each block in command key info (apache#2051) HDDS-5144. Create github check to alert when dependency tree is changed (apache#2177) HDDS-4585. Support bucket acl operation in S3g (apache#1701) HDDS-5153. Decommissioning a dead node should complete immediately (apache#2190) HDDS-5147. Intermittent test failure in TestContainerDeletionChoosingPolicy#testRandomChoosingPolicy (apache#2188) HDDS-5152. Fix Suggested leader in Client. (apache#2189) HDDS-5148. Bump ratis version to 2.1.0-ff8aa66-SNAPSHOT (apache#2184) HDDS-4515. Datanodes should be able to persist and load CRL (apache#2181) HDDS-5060. [SCM HA Security] Make InterSCM grpc channel secure. (apache#2187) HDDS-5051. Ensure failover to suggested leader if any for NotLeaderException. (apache#2141) HDDS-5127. Fix getServiceList when SCM HA is enabled (apache#2173) HDDS-4889. Add simple CI check for docs (apache#2156) HDDS-5131. Use timeout in github actions (apache#2176) HDDS-5103. Fix Install Snapshot Mechanism in SCMStateMachine. (apache#2155) HDDS-5124. Use OzoneConsts.OZONE_TIME_ZONE instead of "GMT" (apache#2166) HDDS-5047. Refactor Pipeline to use ReplicationConfig instead of factor/type (apache#2096) HDDS-5083. Bump version of common-compress (apache#2139) ... Conflicts: hadoop-hdds/common/pom.xml hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineManager.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMStorageConfig.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneClusterImpl.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMStorage.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java

…k-in-auth * HDDS-3698-nonrolling-upgrade: (57 commits) Fix compilation errors afte merge Update javassist in recon pom Fix changes introduced in merge that failed TestSCMNodeManager upgrade tests Fix checkstyle Fix intermittent test failure TestSCMNodeManager#testSetNodeOpStateAndCommandFired after merge Skip scm init default layout version in TestOzoneConfigurationFields HDDS-2212. Genconf tool should generate config files for secure clust… (apache#1788) HDDS-5166. Remove duplicate assignment of OZONE_OPTS for freon and sh (apache#2195) Revert "HDDS-5144. Create github check to alert when dependency tree is changed (apache#2177)" HDDS-4983. Display key offset for each block in command key info (apache#2051) HDDS-5144. Create github check to alert when dependency tree is changed (apache#2177) HDDS-4585. Support bucket acl operation in S3g (apache#1701) HDDS-5153. Decommissioning a dead node should complete immediately (apache#2190) HDDS-5147. Intermittent test failure in TestContainerDeletionChoosingPolicy#testRandomChoosingPolicy (apache#2188) HDDS-5152. Fix Suggested leader in Client. (apache#2189) HDDS-5148. Bump ratis version to 2.1.0-ff8aa66-SNAPSHOT (apache#2184) HDDS-4515. Datanodes should be able to persist and load CRL (apache#2181) HDDS-5060. [SCM HA Security] Make InterSCM grpc channel secure. (apache#2187) HDDS-5051. Ensure failover to suggested leader if any for NotLeaderException. (apache#2141) HDDS-5127. Fix getServiceList when SCM HA is enabled (apache#2173) HDDS-4889. Add simple CI check for docs (apache#2156) HDDS-5131. Use timeout in github actions (apache#2176) HDDS-5103. Fix Install Snapshot Mechanism in SCMStateMachine. (apache#2155) HDDS-5124. Use OzoneConsts.OZONE_TIME_ZONE instead of "GMT" (apache#2166) HDDS-5047. Refactor Pipeline to use ReplicationConfig instead of factor/type (apache#2096) ...

bshashikant force-pushed the HDDS-5051 branch from aac2a00 to 78f7e20Compare April 9, 2021 14:43

bharatviswa504 reviewed Apr 14, 2021
View reviewed changes

bshashikant added the scm-ha label Apr 21, 2021

bshashikant force-pushed the HDDS-5051 branch from 0c9cbfb to 648b813Compare April 21, 2021 09:29

bharatviswa504 reviewed Apr 21, 2021
View reviewed changes

bshashikant force-pushed the HDDS-5051 branch from 9c3a1aa to 6a4fda2Compare April 22, 2021 08:05

bharatviswa504 reviewed Apr 22, 2021
View reviewed changes

Comment thread...src/main/java/org/apache/hadoop/hdds/scm/proxy/SCMSecurityProtocolFailoverProxyProvider.java Outdated

bharatviswa504 reviewed Apr 22, 2021
View reviewed changes

Comment thread...src/main/java/org/apache/hadoop/hdds/scm/proxy/SCMSecurityProtocolFailoverProxyProvider.java Outdated

bharatviswa504 reviewed Apr 22, 2021
View reviewed changes

Comment thread...src/main/java/org/apache/hadoop/hdds/scm/proxy/SCMSecurityProtocolFailoverProxyProvider.java Outdated

bharatviswa504 reviewed Apr 22, 2021
View reviewed changes

Comment thread...src/main/java/org/apache/hadoop/hdds/scm/proxy/SCMSecurityProtocolFailoverProxyProvider.java Outdated

bshashikant added 9 commits April 23, 2021 10:05

HDDS-5051. Ensure failover to suggested leader if any for NotLeaderEx…
b8ebcb2
…ception.

Addressed checkstyle and fndbug issues.
c156895

Addressed checkstyle issues.
5905cff

Addressed checkstyle issues.
45611cc

Addressed findbug issues.
06bbe23

Fixed the retryCount logic based on maxRetryTimeout.
b104584

Addressed checkstyle issues.
9cc09ce

Addressed review comments.
5531bc1

Addressed review comments.
ac68a8d

bshashikant force-pushed the HDDS-5051 branch from b2055f5 to ac68a8dCompare April 23, 2021 04:35

Fixed default scm clinet retry timeout to 30s in docker-configs.
461c1c2

bharatviswa504 reviewed Apr 26, 2021
View reviewed changes

Comment thread...src/main/java/org/apache/hadoop/hdds/scm/proxy/SCMSecurityProtocolFailoverProxyProvider.java

bharatviswa504 reviewed Apr 26, 2021
View reviewed changes

Comment threadhadoop-ozone/dist/src/main/compose/ozonesecure-ha/docker-config

bshashikant added 4 commits April 26, 2021 12:43

Addressed review comments
4b371f4

Added back scm info wait duration config.
05d4a6f

Fixed CS.
b30498f

Fixed default scm clinet timeout to 30s in MiniOzoneCluster.
62f07ae

bharatviswa504 reviewed Apr 26, 2021
View reviewed changes

Fixed cs.
6211c12

symious reviewed Apr 26, 2021
View reviewed changes

Comment threadhadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/proxy/SCMClientConfig.java Outdated

Added missing space character.
d417621

mukul1987 merged commit 1e1eefa into apache:masterApr 26, 2021

Uh oh!

Conversation

bshashikant commented Apr 9, 2021

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

bharatviswa504 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bharatviswa504 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bshashikant commented Apr 21, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bharatviswa504 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bharatviswa504 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants