Skip to content

HBASE-27347 Port FileWatcher from ZK to autodetect keystore/truststore changes in TLS connections - #4869

Merged
wchevreuil merged 6 commits into
apache:masterfrom
anmolnar:HBASE-27347
Nov 11, 2022
Merged

HBASE-27347 Port FileWatcher from ZK to autodetect keystore/truststore changes in TLS connections#4869
wchevreuil merged 6 commits into
apache:masterfrom
anmolnar:HBASE-27347

Conversation

@anmolnar

Copy link
Copy Markdown
Contributor

This patch is the port of ZooKeeper's FileWatcher's functionality which we can take advantage to detect changes in truststore / keystore files for TLS. Cert / key renewal processes don't need HBase services to be restarted with this patch.

cc @bbeaudreault@Apache9

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 30sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗mvndep0m 10sMaven dependency ordering for branch
+1 💚mvninstall3m 32smaster passed
+1 💚compile4m 40smaster passed
+1 💚checkstyle1m 24smaster passed
+1 💚spotless0m 54sbranch has no errors when running spotless:check.
+1 💚spotbugs3m 25smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 10sMaven dependency ordering for patch
+1 💚mvninstall2m 47sthe patch passed
+1 💚compile3m 38sthe patch passed
+1 💚javac3m 38sthe patch passed
+1 💚checkstyle1m 1sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck9m 18sPatch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚spotless0m 41spatch has no errors when running spotless:check.
+1 💚spotbugs3m 36sthe patch passed
_ Other Tests _
+1 💚asflicense0m 23sThe patch does not generate ASF License warnings.
43m 34s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#4869
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux 24d85d40da07 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 41c7bd3
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count79 (vs. ulimit of 30000)
modulesC: hbase-common hbase-client hbase-server U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/1/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@wchevreuilwchevreuil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, overall, just have some minor remarks.

}
// Note: we don't care about delete events
if (shouldResetContext) {
if (LOG.isDebugEnabled()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Maybe worth info logging? My thought here is that this wouldn't be a frequent event, yet a important one to get logged at higher level than debugging.

resetContext.run();
} else {
if (LOG.isDebugEnabled()) {
LOG.debug("Ignoring watch event and keeping previous default SSL context. Event kind: "

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: parameterized logging?

* revision</a>
*/
@InterfaceAudience.Private
public final class FileChangeWatcher {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a straight copy from Zookeeper? Couldn't we just reuse the ZK impl directly, as we already have ZK as a dependency?

@anmolnaranmolnarNov 9, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Urm... I don't have a strong opinion. We were following this pattern in the entire implementation of TLS. It's probably better to stay on the safe side and avoid sideeffects of ZK non-backward compatible changes. They're quite unlikely though.

@wchevreuilwchevreuilNov 10, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for clarifying. Not a big deal for me. And on the flip side, it would be actually a bit weird to depend on ZK for a functionality that isn't really ZK specific, so I'm ok with this copy approach.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 45sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 12sMaven dependency ordering for branch
+1 💚mvninstall2m 33smaster passed
+1 💚compile1m 9smaster passed
+1 💚shadedjars3m 50sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 48smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 11sMaven dependency ordering for patch
+1 💚mvninstall2m 28sthe patch passed
+1 💚compile1m 10sthe patch passed
+1 💚javac1m 10sthe patch passed
+1 💚shadedjars3m 53spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 47sthe patch passed
_ Other Tests _
+1 💚unit1m 56shbase-common in the patch passed.
+1 💚unit1m 17shbase-client in the patch passed.
+1 💚unit193m 26shbase-server in the patch passed.
219m 1s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#4869
Optional Testsjavac javadoc unit shadedjars compile
unameLinux d4cc90e5e458 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 41c7bd3
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/1/testReport/
Max. process+thread count2449 (vs. ulimit of 30000)
modulesC: hbase-common hbase-client hbase-server U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 23sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗mvndep0m 16sMaven dependency ordering for branch
+1 💚mvninstall3m 7smaster passed
+1 💚compile4m 8smaster passed
+1 💚checkstyle1m 10smaster passed
+1 💚spotless0m 49sbranch has no errors when running spotless:check.
+1 💚spotbugs3m 16smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 9sMaven dependency ordering for patch
+1 💚mvninstall2m 41sthe patch passed
+1 💚compile4m 4sthe patch passed
+1 💚javac4m 4sthe patch passed
-0 ⚠️checkstyle0m 16shbase-common: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck10m 35sPatch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚spotless0m 51spatch has no errors when running spotless:check.
+1 💚spotbugs4m 17sthe patch passed
_ Other Tests _
+1 💚asflicense0m 36sThe patch does not generate ASF License warnings.
46m 49s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#4869
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux 2ad66b07bbad 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / da99738
Default JavaEclipse Adoptium-11.0.17+8
checkstylehttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-common.txt
Max. process+thread count80 (vs. ulimit of 30000)
modulesC: hbase-common hbase-client hbase-server U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/2/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 34sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 11sMaven dependency ordering for branch
+1 💚mvninstall2m 59smaster passed
+1 💚compile1m 31smaster passed
+1 💚shadedjars4m 29sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 57smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 10sMaven dependency ordering for patch
+1 💚mvninstall2m 42sthe patch passed
+1 💚compile1m 27sthe patch passed
+1 💚javac1m 27sthe patch passed
+1 💚shadedjars4m 14spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 54sthe patch passed
_ Other Tests _
+1 💚unit2m 1shbase-common in the patch passed.
+1 💚unit1m 21shbase-client in the patch passed.
+1 💚unit212m 29shbase-server in the patch passed.
241m 11s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#4869
Optional Testsjavac javadoc unit shadedjars compile
unameLinux bd1885fef336 5.4.0-131-generic #147-Ubuntu SMP Fri Oct 14 17:07:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / da99738
Default JavaTemurin-1.8.0_352-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/2/testReport/
Max. process+thread count2376 (vs. ulimit of 30000)
modulesC: hbase-common hbase-client hbase-server U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 13sDocker mode activated.
-0 ⚠️yetus0m 5sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 13sMaven dependency ordering for branch
+1 💚mvninstall2m 42smaster passed
+1 💚compile1m 24smaster passed
+1 💚shadedjars3m 47sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc1m 0smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 13sMaven dependency ordering for patch
+1 💚mvninstall2m 36sthe patch passed
+1 💚compile1m 26sthe patch passed
+1 💚javac1m 26sthe patch passed
+1 💚shadedjars3m 46spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 57sthe patch passed
_ Other Tests _
+1 💚unit2m 27shbase-common in the patch passed.
+1 💚unit1m 23shbase-client in the patch passed.
+1 💚unit241m 19shbase-server in the patch passed.
270m 18s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#4869
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 8c48b0875401 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / da99738
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/2/testReport/
Max. process+thread count2484 (vs. ulimit of 30000)
modulesC: hbase-common hbase-client hbase-server U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@wchevreuil
wchevreuil merged commit 047f4e2 into apache:masterNov 11, 2022
@anmolnar
anmolnar deleted the HBASE-27347 branch November 13, 2022 17:39
@anmolnar

Copy link
Copy Markdown
ContributorAuthor

Thanks @wchevreuil for merging the patch. Please cherry pick it to branch-2 as well.

bbeaudreault pushed a commit to HubSpot/hbase that referenced this pull request Nov 16, 2022
…e changes in TLS connections (apache#4869)
Signed-off-by: wchevreuil@apache.org
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@anmolnar@Apache-HBase@wchevreuil