Skip to content

HBASE-24765: Dynamic master discovery - #2314

Merged
bharathv merged 1 commit into
apache:branch-2from
bharathv:dyn_masters_branch-2
Aug 26, 2020
Merged

HBASE-24765: Dynamic master discovery#2314
bharathv merged 1 commit into
apache:branch-2from
bharathv:dyn_masters_branch-2

Conversation

@bharathv

Copy link
Copy Markdown
Contributor

This patch adds the ability to discover newly added masters
dynamically on the master registry side. The trigger for the
re-fetch is either periodic (5 mins) or any registry RPC failure.
Master server information is cached in masters to avoid repeated
ZK lookups.

Updates the client side connection metrics to maintain a counter
per RPC type so that clients have visibility into counts grouped
by RPC method name.

I didn't add the method to ZK registry interface since there
is a design discussion going on in splittable meta doc. We can
add it later if needed.

Signed-off-by: Nick Dimiduk ndimiduk@apache.org
Signed-off-by: Viraj Jasani vjasani@apache.org
Signed-off-by: Duo Zhang zhangduo@apache.org
(cherry picked from commit 275a38e)

This patch adds the ability to discover newly added masters
dynamically on the master registry side. The trigger for the
re-fetch is either periodic (5 mins) or any registry RPC failure.
Master server information is cached in masters to avoid repeated
ZK lookups.
Updates the client side connection metrics to maintain a counter
per RPC type so that clients have visibility into counts grouped
by RPC method name.
I didn't add the method to ZK registry interface since there
is a design discussion going on in splittable meta doc. We can
add it later if needed.
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 275a38e)
@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 34sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗prototool0m 0sprototool was not available.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ branch-2 Compile Tests _
+0 🆗mvndep0m 17sMaven dependency ordering for branch
+1 💚mvninstall4m 20sbranch-2 passed
+1 💚checkstyle2m 24sbranch-2 passed
+1 💚spotbugs7m 31sbranch-2 passed
_ Patch Compile Tests _
+0 🆗mvndep0m 14sMaven dependency ordering for patch
+1 💚mvninstall3m 53sthe patch passed
+1 💚checkstyle0m 9sThe patch passed checkstyle in hbase-protocol-shaded
+1 💚checkstyle0m 35sThe patch passed checkstyle in hbase-client
+1 💚checkstyle0m 13sThe patch passed checkstyle in hbase-zookeeper
+1 💚checkstyle1m 22shbase-server: The patch generated 0 new + 111 unchanged - 1 fixed = 111 total (was 112)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck13m 14sPatch does not cause any errors with Hadoop 3.1.2 3.2.1.
+1 💚hbaseprotoc2m 12sthe patch passed
+1 💚spotbugs7m 51sthe patch passed
_ Other Tests _
+1 💚asflicense0m 44sThe patch does not generate ASF License warnings.
55m 26s
SubsystemReport/Notes
DockerClient=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2314/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#2314
JIRA IssueHBASE-24765
Optional Testsdupname asflicense spotbugs hadoopcheck hbaseanti checkstyle cc hbaseprotoc prototool
unameLinux 31fa6af507d7 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / 5c12b1e
Max. process+thread count84 (vs. ulimit of 12500)
modulesC: hbase-protocol-shaded hbase-client hbase-zookeeper hbase-server U: .
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2314/1/console
versionsgit=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12
Powered byApache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 45sDocker mode activated.
-0 ⚠️yetus0m 7sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+0 🆗mvndep0m 20sMaven dependency ordering for branch
+1 💚mvninstall4m 56sbranch-2 passed
+1 💚compile2m 37sbranch-2 passed
+1 💚shadedjars7m 48sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc1m 46sbranch-2 passed
_ Patch Compile Tests _
+0 🆗mvndep0m 17sMaven dependency ordering for patch
+1 💚mvninstall4m 24sthe patch passed
+1 💚compile2m 52sthe patch passed
+1 💚javac2m 52sthe patch passed
+1 💚shadedjars7m 51spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc2m 21sthe patch passed
_ Other Tests _
+1 💚unit1m 0shbase-protocol-shaded in the patch passed.
+1 💚unit2m 34shbase-client in the patch passed.
+1 💚unit1m 2shbase-zookeeper in the patch passed.
+1 💚unit141m 53shbase-server in the patch passed.
185m 26s
SubsystemReport/Notes
DockerClient=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2314/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR#2314
JIRA IssueHBASE-24765
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 4d7b757b6eaf 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / 5c12b1e
Default Java1.8.0_232
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2314/1/testReport/
Max. process+thread count3426 (vs. ulimit of 12500)
modulesC: hbase-protocol-shaded hbase-client hbase-zookeeper hbase-server U: .
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2314/1/console
versionsgit=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered byApache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 34sDocker mode activated.
-0 ⚠️yetus0m 6sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+0 🆗mvndep0m 19sMaven dependency ordering for branch
+1 💚mvninstall5m 8sbranch-2 passed
+1 💚compile3m 24sbranch-2 passed
+1 💚shadedjars7m 48sbranch has no errors when building our shaded downstream artifacts.
-0 ⚠️javadoc0m 35shbase-client in branch-2 failed.
-0 ⚠️javadoc0m 47shbase-server in branch-2 failed.
-0 ⚠️javadoc0m 19shbase-zookeeper in branch-2 failed.
_ Patch Compile Tests _
+0 🆗mvndep0m 25sMaven dependency ordering for patch
+1 💚mvninstall5m 21sthe patch passed
+1 💚compile3m 14sthe patch passed
+1 💚javac3m 14sthe patch passed
+1 💚shadedjars7m 44spatch has no errors when building our shaded downstream artifacts.
-0 ⚠️javadoc0m 33shbase-client in the patch failed.
-0 ⚠️javadoc0m 19shbase-zookeeper in the patch failed.
-0 ⚠️javadoc0m 47shbase-server in the patch failed.
_ Other Tests _
+1 💚unit1m 5shbase-protocol-shaded in the patch passed.
+1 💚unit2m 50shbase-client in the patch passed.
+1 💚unit0m 53shbase-zookeeper in the patch passed.
+1 💚unit163m 38shbase-server in the patch passed.
209m 6s
SubsystemReport/Notes
DockerClient=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2314/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#2314
JIRA IssueHBASE-24765
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 6c92dc188648 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / 5c12b1e
Default Java2020-01-14
javadochttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2314/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt
javadochttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2314/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
javadochttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2314/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-zookeeper.txt
javadochttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2314/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt
javadochttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2314/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-zookeeper.txt
javadochttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2314/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2314/1/testReport/
Max. process+thread count3778 (vs. ulimit of 12500)
modulesC: hbase-protocol-shaded hbase-client hbase-zookeeper hbase-server U: .
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2314/1/console
versionsgit=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered byApache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@bharathv
bharathv merged commit bb9121d into apache:branch-2Aug 26, 2020
bharathv added a commit to bharathv/hbase that referenced this pull request Sep 7, 2020
This patch adds the ability to discover newly added masters
dynamically on the master registry side. The trigger for the
re-fetch is either periodic (5 mins) or any registry RPC failure.
Master server information is cached in masters to avoid repeated
ZK lookups.
Updates the client side connection metrics to maintain a counter
per RPC type so that clients have visibility into counts grouped
by RPC method name.
I didn't add the method to ZK registry interface since there
is a design discussion going on in splittable meta doc. We can
add it later if needed.
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 275a38e)
(cherry picked from commit bb9121d)
bharathv added a commit to bharathv/hbase that referenced this pull request Sep 8, 2020
This patch adds the ability to discover newly added masters
dynamically on the master registry side. The trigger for the
re-fetch is either periodic (5 mins) or any registry RPC failure.
Master server information is cached in masters to avoid repeated
ZK lookups.
Updates the client side connection metrics to maintain a counter
per RPC type so that clients have visibility into counts grouped
by RPC method name.
I didn't add the method to ZK registry interface since there
is a design discussion going on in splittable meta doc. We can
add it later if needed.
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 275a38e)
(cherry picked from commit bb9121d)
bharathv added a commit to bharathv/hbase that referenced this pull request Sep 8, 2020
This patch adds the ability to discover newly added masters
dynamically on the master registry side. The trigger for the
re-fetch is either periodic (5 mins) or any registry RPC failure.
Master server information is cached in masters to avoid repeated
ZK lookups.
Updates the client side connection metrics to maintain a counter
per RPC type so that clients have visibility into counts grouped
by RPC method name.
I didn't add the method to ZK registry interface since there
is a design discussion going on in splittable meta doc. We can
add it later if needed.
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 275a38e)
(cherry picked from commit bb9121d)
bharathv added a commit to bharathv/hbase that referenced this pull request Sep 16, 2020
This patch adds the ability to discover newly added masters
dynamically on the master registry side. The trigger for the
re-fetch is either periodic (5 mins) or any registry RPC failure.
Master server information is cached in masters to avoid repeated
ZK lookups.
Updates the client side connection metrics to maintain a counter
per RPC type so that clients have visibility into counts grouped
by RPC method name.
I didn't add the method to ZK registry interface since there
is a design discussion going on in splittable meta doc. We can
add it later if needed.
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 275a38e)
(cherry picked from commit bb9121d)
bharathv added a commit to bharathv/hbase that referenced this pull request Sep 16, 2020
This patch adds the ability to discover newly added masters
dynamically on the master registry side. The trigger for the
re-fetch is either periodic (5 mins) or any registry RPC failure.
Master server information is cached in masters to avoid repeated
ZK lookups.
Updates the client side connection metrics to maintain a counter
per RPC type so that clients have visibility into counts grouped
by RPC method name.
I didn't add the method to ZK registry interface since there
is a design discussion going on in splittable meta doc. We can
add it later if needed.
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 275a38e)
(cherry picked from commit bb9121d)
bharathv added a commit to bharathv/hbase that referenced this pull request Sep 18, 2020
This patch adds the ability to discover newly added masters
dynamically on the master registry side. The trigger for the
re-fetch is either periodic (5 mins) or any registry RPC failure.
Master server information is cached in masters to avoid repeated
ZK lookups.
Updates the client side connection metrics to maintain a counter
per RPC type so that clients have visibility into counts grouped
by RPC method name.
I didn't add the method to ZK registry interface since there
is a design discussion going on in splittable meta doc. We can
add it later if needed.
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 275a38e)
(cherry picked from commit bb9121d)
bharathv added a commit that referenced this pull request Sep 19, 2020
This patch adds the ability to discover newly added masters
dynamically on the master registry side. The trigger for the
re-fetch is either periodic (5 mins) or any registry RPC failure.
Master server information is cached in masters to avoid repeated
ZK lookups.
Updates the client side connection metrics to maintain a counter
per RPC type so that clients have visibility into counts grouped
by RPC method name.
I didn't add the method to ZK registry interface since there
is a design discussion going on in splittable meta doc. We can
add it later if needed.
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
(cherry picked from commit 275a38e)
(cherry picked from commit bb9121d)
clarax pushed a commit to clarax/hbase that referenced this pull request Nov 15, 2020
This patch adds the ability to discover newly added masters
dynamically on the master registry side. The trigger for the
re-fetch is either periodic (5 mins) or any registry RPC failure.
Master server information is cached in masters to avoid repeated
ZK lookups.
Updates the client side connection metrics to maintain a counter
per RPC type so that clients have visibility into counts grouped
by RPC method name.
I didn't add the method to ZK registry interface since there
is a design discussion going on in splittable meta doc. We can
add it later if needed.
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 275a38e)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@bharathv@Apache-HBase@Apache9