HDDS-5246. Wait for ever to obtain CA list which is needed during OM/DN startup by bharatviswa504 · Pull Request #2266 · apache/ozone

bharatviswa504 · 2021-05-19T09:07:33Z

What changes were proposed in this pull request?

Wait forever to obtain the CA list.
And also made retry wait configurable.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5246

How was this patch tested?

Tested with docker-compose ozonesecure-ha (with few changes to remove WAITFOR scm3.org)

docker-compose up scm1.org
docker-compose om1
docker-compose datanode1

om1_1 | 2021-05-19 09:02:16,082 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
om1_1 | 2021-05-19 09:02:26,102 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
om1_1 | 2021-05-19 09:02:26,103 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
om1_1 | 2021-05-19 09:02:36,147 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
om1_1 | 2021-05-19 09:02:36,147 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
om1_1 | 2021-05-19 09:02:46,198 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
om1_1 | 2021-05-19 09:02:46,198 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
om1_1 | 2021-05-19 09:02:56,222 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
om1_1 | 2021-05-19 09:02:56,222 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
om1_1 | 2021-05-19 09:03:06,245 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
om1_1 | 2021-05-19 09:03:06,245 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
om1_1 | 2021-05-19 09:03:16,263 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
om1_1 | 2021-05-19 09:03:16,263 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
om1_1 | 2021-05-19 09:03:26,296 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
om1_1 | 2021-05-19 09:03:26,296 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
om1_1 | 2021-05-19 09:03:36,333 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 3.
om1_1 | 2021-05-19 09:03:36,333 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
om1_1 | 2021-05-19 09:03:46,376 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 3.
om1_1 | 2021-05-19 09:03:46,376 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms

datanode1_1 | 2021-05-19 09:02:50,155 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
datanode1_1 | 2021-05-19 09:03:00,179 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
datanode1_1 | 2021-05-19 09:03:00,179 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
datanode1_1 | 2021-05-19 09:03:10,204 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
datanode1_1 | 2021-05-19 09:03:10,204 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
datanode1_1 | 2021-05-19 09:03:20,225 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
datanode1_1 | 2021-05-19 09:03:20,225 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
datanode1_1 | 2021-05-19 09:03:30,266 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
datanode1_1 | 2021-05-19 09:03:30,266 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
datanode1_1 | 2021-05-19 09:03:40,293 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 3.
datanode1_1 | 2021-05-19 09:03:40,293 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
datanode1_1 | 2021-05-19 09:03:50,323 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 3.
datanode1_1 | 2021-05-19 09:03:50,323 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms

bshashikant

looks good

adoroszlai

LGTM, but I'd like to suggest some some trivial code improvements.

…ls/HAUtils.java Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

…ls/HAUtils.java Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

bharatviswa504 · 2021-05-19T11:32:06Z

Thank You @adoroszlai for the thorough review.
Addressed review comments.

adoroszlai

Thanks @bharatviswa504 for updating the patch.

xiaoyuyao · 2021-05-19T15:40:52Z


+
+ <property>
+ <name>ozone.scm.ca.list.retry.wait.duration</name>


NIT: Can we rename ozone.scm.ca.list.retry.wait.duration to ozone.scm.ca.list.retry.interval to be consistent with other retry settings?

xiaoyuyao

LGTM, I just have one minor suggestion added inline...

xiaoyuyao

+1 pending CI.

…DN startup (apache#2266) (cherry picked from commit 08375d7) Change-Id: I2aa46fc70ec37059946da1c69806bc09bbf4f0ae

bharatviswa504 requested review from bshashikant and xiaoyuyao May 19, 2021 09:07

bharatviswa504 added the scm-ha label May 19, 2021

retry for ever to obtain ca list
1fe5c7c

bharatviswa504 force-pushed the HDDS-5246 branch from 34a113d to 1fe5c7cCompare May 19, 2021 09:44

bshashikant approved these changes May 19, 2021
View reviewed changes

adoroszlai requested changes May 19, 2021
View reviewed changes

bharatviswa504and others added 8 commits May 19, 2021 16:04

Update hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/uti…
a31faa6
…ls/HAUtils.java Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

Update hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/uti…
f20f025
…ls/HAUtils.java Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

Update hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/uti…
c0d36a2
…ls/HAUtils.java Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

Update hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/uti…
6bc0582
…ls/HAUtils.java Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

Update hadoop-hdds/common/src/main/resources/ozone-default.xml
e3ccdc2
Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

Update hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/uti…
d47a7c7
…ls/HAUtils.java Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

Update hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/uti…
80d69db
…ls/HAUtils.java Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

review comments
fa5f5c7

bharatviswa504 requested a review from adoroszlai May 19, 2021 11:32

adoroszlai approved these changes May 19, 2021
View reviewed changes

xiaoyuyao reviewed May 19, 2021
View reviewed changes

fix review comment
feccff4

bharatviswa504 requested a review from xiaoyuyao May 19, 2021 16:26

xiaoyuyao approved these changes May 19, 2021
View reviewed changes

mukul1987 merged commit 08375d7 into apache:masterMay 20, 2021

Navigation Menu

Uh oh!

HDDS-5246. Wait for ever to obtain CA list which is needed during OM/DN startup - #2266

HDDS-5246. Wait for ever to obtain CA list which is needed during OM/DN startup#2266
mukul1987 merged 10 commits into
apache:masterfrom
bharatviswa504:HDDS-5246

bharatviswa504 commented May 19, 2021

Uh oh!

bshashikant left a comment

Uh oh!

adoroszlai left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bharatviswa504 commented May 19, 2021

Uh oh!

adoroszlai left a comment

Uh oh!

xiaoyuyaoMay 19, 2021

Uh oh!

bharatviswa504May 19, 2021

Uh oh!

xiaoyuyao left a comment

Uh oh!

xiaoyuyao left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants



		<property>
		<name>ozone.scm.ca.list.retry.wait.duration</name>

Uh oh!

Conversation

bharatviswa504 commented May 19, 2021

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

bshashikant left a comment

Choose a reason for hiding this comment

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bharatviswa504 commented May 19, 2021

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

xiaoyuyaoMay 19, 2021

Choose a reason for hiding this comment

Uh oh!

bharatviswa504May 19, 2021

Choose a reason for hiding this comment

Uh oh!

xiaoyuyao left a comment

Choose a reason for hiding this comment

Uh oh!

xiaoyuyao left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants