Uh oh!
There was an error while loading. Please reload this page.
HBASE-26666 Add native TLS encryption support to RPC server/client - #4125
Closed
anmolnar wants to merge 2 commits into
Closed
HBASE-26666 Add native TLS encryption support to RPC server/client#4125anmolnar wants to merge 2 commits into
anmolnar wants to merge 2 commits into
Conversation
Apache-HBase
commented
Feb 23, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Feb 23, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Feb 23, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Feb 25, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Feb 25, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Feb 25, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Feb 25, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Feb 25, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Feb 25, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Feb 28, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Feb 28, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Feb 28, 2022
💔 -1 overall
This message was automatically generated. |
anmolnarforce-pushed
the
HBASE-26666_master
branch
from
March 1, 2022 16:34
cef6e3b to
4debffcCompareApache-HBase
commented
Mar 1, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Mar 1, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Mar 1, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Mar 1, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Mar 1, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Mar 1, 2022
🎊 +1 overall
This message was automatically generated. |
anmolnar
marked this pull request as ready for review
March 2, 2022 12:43
anmolnarforce-pushed
the
HBASE-26666_master
branch
from
March 2, 2022 17:48
5996d63 to
8fe9b62CompareApache-HBase
commented
Mar 2, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Mar 3, 2022
💔 -1 overall
This message was automatically generated. |
taklwu
reviewed
Mar 8, 2022
Contributor
There was a problem hiding this comment.
LGTM. please give another try and see if we can have a clean build.
[nit] I may miss something but I'm wondered if we have any test or manual that has use HBASE_NETTY_RPCSERVER_TLS_ENABLED=true ? this is not a blocker, and I think adding the ssl header should be as straightforward as it tested in TestX509Util
Uh oh!
There was an error while loading. Please reload this page.
wchevreuil
left a comment
Contributor
There was a problem hiding this comment.
Overall, LGTM. Have you reviewed all latest UT failures? Could those be related?
meszibalu
requested changes
Mar 10, 2022
| } | ||
| } | ||
| }).channel(); | ||
| }).sync().channel(); |
Contributor
There was a problem hiding this comment.
Why do we have sync() here? It converts the asynchronous connection to synchronous.
ContributorAuthor
There was a problem hiding this comment.
HBase shell is not able to connect without this. I'm sure I've seen errors in RS-Master communication as well, but it works in most cases.
The issue is:
ERROR: Failed contacting masters after 1 attempts.
Exceptions:
java.io.IOException: Call to address=Andors-MacBook-Pro.local:16000 failed on local exception:
java.io.IOException: java.lang.UnsupportedOperationException: unsupported message type: Call (expected: ByteBuf, FileRegion)
My thinking is that connect() is having a race with sendRequest(), becaue NettyRpcConnection is trying to establish the connection in lazy way: doesn't do anything until the first request (Call) comes in. Because of this we have to make sure that the Netty pipeline is in a usable state before letting sendRequest() to push the Call into and seems like sync() is enough. It waits for the connection to be established before we give back the channel.
This is still not ideal IMHO: we could implement a latch to wait for the SSL handshake to happen before doing anything else - like ZooKeeper does -, but it might be overkill in this case given that it already works fine with sync().
Member
There was a problem hiding this comment.
Your explanation certainly makes sense to me. I think keeping the Channel async is probably going to give us some performance gains.
If this only affected the hbase client when TLS was on, it's ok to leave this sync() for now and come back to it later. However, this is going to affect all clients so we may be introducing a performance regression for this optional feature which isn't good.
You mentioned a latch for when the TLS handshake is done. I vaguely remember a similar latch for the SASL handshake (though, it might have be just a synchronous check rather than asynchronous). It's also OK for us to commit this patch onto the feature branch and address it later. We just have to remember that this would block our merge back to master.
Contributor
There was a problem hiding this comment.
There is a BufferCallBeforeInitHandler to handle the case for writing rpc call out before initializing the connection.
That's why we can avoid calling sync in this method.
ContributorAuthor
There was a problem hiding this comment.
@Apache9 Yep, I saw that class piles up the Calls until it receives the green signal, but for some reason SSL doesn't work if the handler precedes the SSL handler. I'll take another look.
@joshelser I'd like to commit this independently from the JWT feature if it's feasible. The PR is against the master branch intentionally.
Contributor
There was a problem hiding this comment.
In the netty world, I think you should use callback instead of block waiting. If a feature can be implemented by a sync call, I think it should also be possible to be implemented by a ChannelFuture.addListener or something else.
Comment on lines
+325
to
+326
| channelRef.compareAndSet(null, connect()); | ||
| channel = channelRef.get(); |
Contributor
There was a problem hiding this comment.
You discarded the return value of compareAndSet. If there are multiple connection attempts at the same time, one of it is leaked.
| if (in.readableBytes() >= 5) { | ||
| super.decode(context, in, out); | ||
| } else if (in.readableBytes() > 0) { | ||
| // It requires 5 bytes to detect a proper ssl connection. In the |
Contributor
There was a problem hiding this comment.
If 5 bytes are required why >= 5 in the previous if?
ContributorAuthor
There was a problem hiding this comment.
That's copied from the base class.
Comment on lines
+247
to
+250
| if (TLS_HANDSHAKE_RECORD_TYPE != in.getByte(0)) { | ||
| LOG.debug("first byte {} does not match TLS handshake, failing to plaintext", | ||
| in.getByte(0)); | ||
| handleNonSsl(context); |
Contributor
There was a problem hiding this comment.
It reconfigures the pipeline once if we have only 1-4 incoming bytes. So if at some point of time we receive only 2 bytes of data for example, it will remove itself and the ssl handler if I understand it correctly.
I think we should check the first X bytes of incoming data and make the decision to keep or remove ssl handler.
ContributorAuthor
There was a problem hiding this comment.
I ended up removing this class. That's something which was needed for ZooKeeper 4LW commands which is obviously not the case for HBase.
anmolnar
commented
Mar 17, 2022
ContributorAuthor
@meszibalu Please take another look. I addressed your comments. |
anmolnar
commented
Mar 17, 2022
ContributorAuthor
Out of curiosity, does anybody know why HBase not using Epoll (native transport) for client sockets? |
anmolnar
commented
Mar 17, 2022
ContributorAuthor
@taklwu Yes, I think an e2e test would make perfect sense here like we did for the JWT. We might also need a combined test for encryption + JWT. Let me work on this. |
Apache-HBase
commented
Mar 17, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 21, 2022
🎊 +1 overall
This message was automatically generated. |
anmolnarforce-pushed
the
HBASE-26666_master
branch
from
July 25, 2022 10:19
111f518 to
5b0cc1cCompareContributorAuthor
@bbeaudreault I've squased and rebased the patch. |
Apache-HBase
commented
Jul 25, 2022
💔 -1 overall
This message was automatically generated. |
anmolnar
commented
Jul 25, 2022
ContributorAuthor
What's wrong with this build again? |
Apache-HBase
commented
Jul 25, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 25, 2022
🎊 +1 overall
This message was automatically generated. |
anmolnar
commented
Jul 25, 2022
ContributorAuthor
Okay, two +1, one -1 in the latest iteration. What should I do? Shall I restart and hope the best? |
| public static final String TLS_CONFIG_CLR = CONFIG_PREFIX + "clr"; | ||
| public static final String TLS_CONFIG_OCSP = CONFIG_PREFIX + "ocsp"; | ||
| public static String HBASE_CLIENT_NETTY_TLS_ENABLED = "hbase.client.netty.tls.enabled"; |
Contributor
There was a problem hiding this comment.
i think you need to change this and below to static final -- i think that's why checkstyle is complaining
bbeaudreault
commented
Jul 26, 2022
Contributor
i asked on slack about the failure. unfortunately I dont really know. one thing I notice is the failure is actually for while waiting, I also noticed 2 small things. one of which is contributing to the warnings above. not quite sure what's up with the Why don't you fix those 2 small things and then we'll give someone in slack a day to check in and if no one has any ideas we can look to merge. |
Apache9
commented
Jul 26, 2022
Contributor
Not sure what's the real problem, we do not touch the code in the PR here and ImmutableEnumCheck is not a blocker which should not fail compilation I believe. I tried locally, I could build the master branch without any failure. |
Apache-HBase
commented
Jul 26, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 26, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 26, 2022
🎊 +1 overall
This message was automatically generated. |
meszibalu
commented
Jul 27, 2022
Contributor
We cannot merge this commit, because it conflicts with PR 4624. |
anmolnarforce-pushed
the
HBASE-26666_master
branch
from
July 27, 2022 08:31
6ca13b4 to
6c9dbd0Compareanmolnar
commented
Jul 27, 2022
ContributorAuthor
I'm done with the rebase, there was only one conflicting file. Let's hope the best with the build. |
Apache-HBase
commented
Jul 27, 2022
🎊 +1 overall
This message was automatically generated. |
Apache9
requested changes
Jul 27, 2022
Apache9
left a comment
Contributor
There was a problem hiding this comment.
I do not think this PR is qualified enough to commit.
When reviewing the first class(NettyRpcConnection), I saw a bunch of changes which I can not understand...
Please hold on committing this...
| /** | ||
| * RPC connection implementation based on netty. | ||
| * <p/> |
| channel.pipeline().fireUserEventTriggered(new CallEvent(TIMEOUT, call)); | ||
| } | ||
| }); | ||
| Channel channel = getChannel(); |
| } | ||
| private void established(Channel ch) throws IOException { | ||
| assert eventLoop.inEventLoop(); |
Contributor
There was a problem hiding this comment.
Why removing this assert? I'm totally confused why we need to change so much assumptions on the implementation of this class? Is it really necessary?
Apache9
commented
Jul 27, 2022
Contributor
Seems the works need to be done here is very straight forward? Check the configuration, if TLS is enabled, we add a SslHandler to the pipeline, it will handle all the things and we even do not need to care whether the handshake is finished before writing to the channel? Then I really do not understand why we need so much changes on the client side NettyRpcConnection? Just use a new ChannelInitializer to add a SslHandler and then a BufferCallBeforeInitHandler is enough? And I also do not understand why we need to synchronized on the initSSL method, both server side and client side? The method will only be called inside the event loop, there is no race at all... |
Apache9
reviewed
Jul 27, 2022
| } | ||
| public SslContext createNettyJdkSslContext(SSLContext sslContext, boolean isClientSocket) { | ||
| return new JdkSslContext(sslContext, isClientSocket, cipherSuitesAsList, |
Contributor
There was a problem hiding this comment.
Please use SslContextBuilder instead of hard coded JdkSslContext here.
SslContextBuilder will try to use open ssl if possible, where the performance is much better than the jdk one.
| return channelsCount > 0 ? channelsCount - 1 : channelsCount; | ||
| } | ||
| private synchronized void initSSL(ChannelPipeline p, boolean supportPlaintext) |
Contributor
There was a problem hiding this comment.
Oh, this one is shared in NettyRpcServer, not per connection, so synchronized is needed if we share some common fields which are not thread safe(but seems not...)
I think we should share the SslContext stuff? That means, when constructing the NettyRpcServer, we check whether TLS is enabled, if so, we create the SslContext instance. So then when initializing a Channel, we just need to call the createSSLEngine method, which should be thread safe so we do not need synchronized here.
| // Basically we only need to create it once. | ||
| private synchronized void initSSL(ChannelPipeline pipeline) | ||
| throws X509Exception.SSLContextException { | ||
| if (sslContext == null || sslEngine == null) { |
Contributor
There was a problem hiding this comment.
I think here we should share the SslContext instance. That means, we'd better create a SslContext instance per RpcClient, and pass it in when creating a NettyRpcConnection, and then just call its createSSLEngine method when initializing a Channel, where we do not need synchronized any more.
Apache-HBase
commented
Jul 27, 2022
💔 -1 overall
This message was automatically generated. |
Apache9
commented
Jul 27, 2022
Contributor
Apache9
commented
Jul 27, 2022
Contributor
With this approach I could get the TestTLSIPC pass. But probably we should try to add the SslHandler before connected if we want to support tcp fast open in the future. Will dig more on why adding before connected will hang. And I noticed that, in the X509Util you create a javax's SSLContext, instead of netty's SslContext, this is not good. You should try to use netty's SslContextBuilder to create a SslContext as netty will try its best to make use of open ssl if possible, which has a much better performance. Thanks. |
Apache-HBase
commented
Jul 27, 2022
💔 -1 overall
This message was automatically generated. |
Apache9
commented
Jul 27, 2022
Contributor
OK, I found the problem. The problem is in BufferCallBeforeInitHandler, we just stop the propagation of the write event, but still propagate the flush event, this messes up the logic in SslHandler so the initialization is hang. Please check the newest commit here I ran TestTLSIPC and TestTlsWithKerberos(why a TLS and then a Tls?) locally, they could both pass. Thanks. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HBASE-26666 Address bearer token being sent over wire before RPC encryption is enabled
because it's needed for JWT authentication. The bearer token must be transmitted on an encrypted channel.
https://issues.apache.org/jira/browse/ZOOKEEPER-2120
Still working on unit tests and documentation.