Uh oh!
There was an error while loading. Please reload this page.
HBASE-25071 ReplicationServer support start ReplicationSource internal - #2452
Merged
Conversation
infraioforce-pushed
the
HBASE-25071-new
branch
3 times, most recently
from
September 24, 2020 10:18
1edb685 to
8e6639eCompareApache-HBase
commented
Sep 24, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Sep 24, 2020
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Sep 24, 2020
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Sep 24, 2020
🎊 +1 overall
This message was automatically generated. |
| message ListReplicationSinkServersResponse { | ||
| repeated ServerName server_name = 1; | ||
| required bool is_replication_server = 1; |
Contributor
There was a problem hiding this comment.
Is this right name for the attribute? The method name for a boolean should be isATTRIBUTE but the ATTRIBUTE itself should not have the 'is' prefix.
Contributor
There was a problem hiding this comment.
Oh, you can't change the protobuf index.... it breaks compatibility. Give the replication_server '2' and leave server_name as '1'.
ContributorAuthor
There was a problem hiding this comment.
This is only for the feature branch. Not merged to any branch now.
Uh oh!
There was an error while loading. Please reload this page.
| private boolean fetchServersUseZk = false; | ||
| private FetchServersChore fetchServersChore; | ||
| private int shortOperationTimeout; | ||
| private boolean isReplicationServer = false; |
Contributor
There was a problem hiding this comment.
Yeah, boolean should be named replicationServer and the method to access this data member is named isReplicationServer.
| .createBlockingRpcChannel(master, User.getCurrent(), shortOperationTimeout)); | ||
| ListReplicationSinkServersResponse resp = masterStub | ||
| .listReplicationSinkServers(null, ListReplicationSinkServersRequest.newBuilder().build()); | ||
| isReplicationServer = resp.getIsReplicationServer(); |
Contributor
There was a problem hiding this comment.
Do you have to do hasReplicationServer first? And then read it if present? Or maybe this defaults false if not present?
ContributorAuthor
There was a problem hiding this comment.
This field is required. So not need to check hasReplicationServer first?
| @InterfaceAudience.Private | ||
| @SuppressWarnings({ "deprecation"}) | ||
| public class HReplicationServer extends Thread implements Server { | ||
| public class HReplicationServer extends Thread implements Server, ReplicationSourceController { |
Contributor
There was a problem hiding this comment.
Looks like ReplicationServer already committed (why have the 'H' prefix?)
| private UserProvider userProvider; | ||
| protected final ReplicationServerRpcServices rpcServices; | ||
| private final ReplicationServerRpcServices rpcServices; |
Contributor
There was a problem hiding this comment.
We have to do this? Just because we do it for HMaster and HRegionServer, it doesn't make it a good pattern. This is an Interface to pass instead of the impl?
ContributorAuthor
There was a problem hiding this comment.
I am ok with this. Decouple a RpcService can make the code more clearly.
| private final ConcurrentMap<String, ReplicationSourceInterface> sources = new ConcurrentHashMap<>(); | ||
| private final ReplicationQueueStorage queueStorage; | ||
| private final ReplicationPeers replicationPeers; |
Contributor
There was a problem hiding this comment.
Is this stuff normally inside a ReplicationSourceManager? Or maybe the Replication instance? Can you not use these instead?
ContributorAuthor
There was a problem hiding this comment.
The key thing of this feature is here. The new HReplicationServer will be a independent server which manage some ReplicationSource.
The assumption here is that: Replication is a producer-consumer module. The "producer" RegionServer write WAL to HDFS and write WAL name to a replication queue, which can be identified by <RegionServer ServerName, QueueId>. The "consumer" ReplicationSource fetch the WAL name from the replication queue and read the WAL from HDFS and replicate to other HBase Clusters or other Systems.
When replication offload enabled, ReplicationSourceManager will only wirte the WAL name to the replication queue. But not start any ReplicationSource. HMaster get all replication queues and assign them to many ReplicationServers. Same with the region balancer, HMaster called a StartReplicationSource rpc to the ReplicationServer. Then the ReplicationServer will start a ReplicationSource thread to do the replication job.
Uh oh!
There was an error while loading. Please reload this page.
| * @param queueStorage the replication queue storage | ||
| * @param replicationPeer the replication peer | ||
| * @param server the server which start and run this replication source | ||
| * @param producer the name of region server which produce the replication queue |
Contributor
There was a problem hiding this comment.
This is for the RecoveryReplicationSource only?
ContributorAuthor
There was a problem hiding this comment.
For all ReplicationSource. The assumption here is that: all replication is a producer-consumer module.
| // init replication source | ||
| src.init(conf, fs, logDir, this, queueStorage, replicationPeer, server, queueId, clusterId, | ||
| walFileLengthProvider, metrics); | ||
| src.init(conf, fs, logDir, this, queueStorage, replicationPeer, server, server.getServerName(), |
Contributor
There was a problem hiding this comment.
There will be a time when the passed in 'server' differs from server.getServerName?
Contributor
There was a problem hiding this comment.
Perhaps say more about this?
ContributorAuthor
ContributorAuthor
There was a problem hiding this comment.
One replication queue can be identified by <Producer ServerName, QueueId>.
ContributorAuthor
There was a problem hiding this comment.
And one ReplicationSource is responsible for one replication queue.
ContributorAuthor
There was a problem hiding this comment.
When replication offload disabled, the ReplicationSource will be started in RegionServer inside. So the producer name is same with the Server's name.
infraioforce-pushed
the
HBASE-25071-new
branch
from
September 25, 2020 01:12
8e6639e to
b7e51dbCompareApache-HBase
commented
Sep 25, 2020
🎊 +1 overall
This message was automatically generated. |
infraioforce-pushed
the
HBASE-25071-new
branch
from
September 25, 2020 02:38
b7e51db to
425ffb4CompareApache-HBase
commented
Sep 25, 2020
💔 -1 overall
This message was automatically generated. |
infraioforce-pushed
the
HBASE-25071-new
branch
from
September 25, 2020 08:23
425ffb4 to
bb46ef9CompareApache-HBase
commented
Sep 25, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Sep 25, 2020
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Sep 25, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 2, 2020
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 2, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 2, 2020
💔 -1 overall
This message was automatically generated. |
infraioforce-pushed
the
HBASE-25071-new
branch
2 times, most recently
from
November 2, 2020 11:12
8a063c3 to
108ddddCompareApache-HBase
commented
Nov 2, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 2, 2020
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 2, 2020
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 2, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 2, 2020
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 2, 2020
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 4, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 4, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 4, 2020
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 4, 2020
🎊 +1 overall
This message was automatically generated. |
ddupg
reviewed
Nov 4, 2020
| if (conf.getBoolean(HConstants.REPLICATION_OFFLOAD_ENABLE_KEY, | ||
| HConstants.REPLICATION_OFFLOAD_ENABLE_DEFAULT)) { | ||
| fetchWALsThread = new Thread(() -> { |
Member
There was a problem hiding this comment.
interrrupt the fetchWALsThread when the ReplicationServer exits or when the peer terminates?
Apache-HBase
commented
Nov 4, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 4, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 5, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 5, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 5, 2020
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 6, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 6, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 6, 2020
💔 -1 overall
This message was automatically generated. |
infraio added a commit
that referenced
this pull request
Nov 9, 2020
#2452) Signed-off-by: XinSun <ddupgs@gmail.com>
ddupg pushed a commit
that referenced
this pull request
Feb 23, 2021
#2452) Signed-off-by: XinSun <ddupgs@gmail.com>
ddupg pushed a commit
that referenced
this pull request
Jul 9, 2021
#2452) Signed-off-by: XinSun <ddupgs@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.