Uh oh!
There was an error while loading. Please reload this page.
HBASE-27217 Revisit the DumpReplicationQueues tool - #4810
Merged
Apache9 merged 2 commits intoNov 13, 2022
Conversation
Apache-HBase
commented
Oct 2, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Oct 2, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Oct 2, 2022
🎊 +1 overall
This message was automatically generated. |
2005hithlj
commented
Oct 3, 2022
ContributorAuthor
@Apache9 sir. Could you take a look? Thanks. |
Apache9
reviewed
Oct 4, 2022
Apache9
left a comment
Contributor
There was a problem hiding this comment.
Left some comments, PTAL.
Thanks @2005hithlj !
| } | ||
| } | ||
| public String dumpReplicationViaTable(Connection connection) throws ReplicationException { |
| if (queueId.isRecovered()) { | ||
| wals = AbstractFSWALProvider | ||
| .getArchivedWALFiles(connection.getConfiguration(), queueId.getSourceServerName().get(), | ||
| URLEncoder.encode(queueId.getSourceServerName().get().toString(), |
| sb.append(" PeerID: " + queueId.getPeerId() + "\n"); | ||
| sb.append(" Recovered: " + queueId.isRecovered() + "\n"); | ||
| if (queueId.getSourceServerName().isPresent()) { | ||
| sb.append(" Dead RegionServers: " + queueId.getSourceServerName().get() + "\n"); |
Contributor
There was a problem hiding this comment.
Actually here we only record the first region server instead of the all the region servers, so maybe we should change the wording here in the future...
| // long position = queueStorage.getWALPosition(regionserver, queueInfo.getPeerId(), wal); | ||
| // sb.append(" Replication position for " + wal + ": " | ||
| // + (position > 0 ? position : "0" + " (not started or nothing to replicate)") + "\n"); | ||
| long position = -1; |
Contributor
There was a problem hiding this comment.
I think here we should filter out the wals before the offset?
| position = offset.getOffset(); | ||
| } | ||
| } | ||
| sb.append(" Replication position for " + (walGroup != null ? walGroup + "/" + wal : wal) |
Contributor
There was a problem hiding this comment.
Notice that, -1 in ReplicationGroupOffset means this file has already been fully replicated.
Apache9force-pushed
the
HBASE-27109/table_based_rqs
branch
2 times, most recently
from
October 12, 2022 07:55
82a795a to
67a94a6CompareApache9
reviewed
Oct 19, 2022
| Set<String> peerIds = | ||
| peers.stream().map((peer) -> peer.getPeerId()).collect(Collectors.toSet()); | ||
| System.out.println(dumpQueues(zkw, peerIds, opts.isHdfs())); | ||
| System.out.println(dumpQueues(zkw, connection, peerIds, opts.isHdfs())); |
| for (ReplicationQueueId queueId : queueIds) { | ||
| List<String> wals = null; | ||
| if (queueId.isRecovered()) { | ||
| wals = AbstractFSWALProvider |
Contributor
There was a problem hiding this comment.
Just use queueId.getServerWALsBelongTo?
ContributorAuthor
There was a problem hiding this comment.
This method can really solve my problem.
| queueId.getSourceServerName().get().toString()) | ||
| .stream().map(Path::toString).collect(Collectors.toList()); | ||
| } else { | ||
| wals = AbstractFSWALProvider |
Contributor
There was a problem hiding this comment.
Is this enough to get all the wals for a live region server?
| StringBuilder sb = new StringBuilder(); | ||
| ReplicationQueueStorage queueStorage = | ||
| ReplicationStorageFactory.getReplicationQueueStorage(connection, getConf()); | ||
| Set<ServerName> liveRegionServers = ZKUtil.listChildrenNoWatch(zkw, zkw.getZNodePaths().rsZNode) |
Contributor
There was a problem hiding this comment.
This could also be done through Admin interface, so we do not need to depend on zk?
| queueId.getServerName().toString()) | ||
| .stream().map(Path::toString).collect(Collectors.toList()); | ||
| } | ||
| Collections.sort(wals); |
Contributor
There was a problem hiding this comment.
String compare is enough for sorting? Not sure...
Uh oh!
There was an error while loading. Please reload this page.
Apache-HBase
commented
Oct 19, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Oct 19, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Oct 19, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Oct 22, 2022
🎊 +1 overall
This message was automatically generated. |
Apache9
reviewed
Oct 24, 2022
| } | ||
| } | ||
| private String dumpReplicationViaTable(Connection connection) throws ReplicationException { |
Contributor
There was a problem hiding this comment.
Checked the code in ZKDump.getReplicationZnodesDump, I do not think this is a direct mapping of the code there?
In ZKDump.getReplicationZnodesDump, first we dump the replication peer data, while here we do not dump it.
And when dumping the replication queue data, we will iterate all rsZNode and print them all, but here we first print all the peers and region servers, and then print all the wal files? Seems a bit strange...
And for dumping hfile refs, we will print hfile refs for each peer in ZKDump.getReplicationZnodesDump, but here we just print them all, without the peer information...
| queueId.getServerName().toString()) | ||
| .stream().map(Path::toString).collect(Collectors.toList()); | ||
| } | ||
| Collections.sort(wals); |
| " Replication position for " + (walGroup != null ? walGroup + "/" + wal : wal) + ": "); | ||
| // Position is -1, which means that the file has already been fully replicated, | ||
| // the logic here is different from the previous version. | ||
| if (position == -1) { |
Contributor
There was a problem hiding this comment.
Is it possible that we arrive here but position is -1? I think it will be filtered out in the above ReplicationOffsetUtil.shouldReplicate check?
| peerIds.add("1"); | ||
| dumpQueues.setConf(config); | ||
| String dump = dumpQueues.dumpQueues(zkWatcherMock, peerIds, false); | ||
| String dump = dumpQueues.dumpQueues(null, peerIds, false); |
Contributor
There was a problem hiding this comment.
Parsing null is enough here? We do not need to pass a Connection instance?
Apache9force-pushed
the
HBASE-27109/table_based_rqs
branch
from
October 24, 2022 11:44
67a94a6 to
6d4e7fdCompareApache9force-pushed
the
HBASE-27109/table_based_rqs
branch
from
November 1, 2022 04:29
6d4e7fd to
56f0241CompareApache9
commented
Nov 1, 2022
Contributor
I've done a rebase on HBASE-27109, please also rebase here. Thanks. @2005hithlj |
2005hithlj
commented
Nov 2, 2022
ContributorAuthor
@Apache9 OK sir, and I will have a new commit for the comments. |
Apache-HBase
commented
Nov 4, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 4, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 4, 2022
🎊 +1 overall
This message was automatically generated. |
Apache9
reviewed
Nov 5, 2022
Uh oh!
There was an error while loading. Please reload this page.
| } | ||
| } | ||
| // only for test | ||
| if (tmpWals.isEmpty() && !wals.contains(offset.getWal())) { |
Uh oh!
There was an error while loading. Please reload this page.
Apache9force-pushed
the
HBASE-27109/table_based_rqs
branch
2 times, most recently
from
November 6, 2022 09:13
7d1cbb3 to
c5196d7CompareApache9
commented
Nov 6, 2022
Contributor
HBASE-27218 has been merged, so you need a rebase here... |
2005hithlj
commented
Nov 7, 2022
ContributorAuthor
OK, sir. |
2005hithljforce-pushed
the
HBASE-27217
branch
2 times, most recently
from
November 7, 2022 16:39
e90d2c9 to
e9cbb69CompareApache-HBase
commented
Nov 12, 2022
🎊 +1 overall
This message was automatically generated. |
Apache9 pushed a commit
that referenced
this pull request
Nov 13, 2022
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
Nov 27, 2022
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
Nov 30, 2022
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
Dec 3, 2022
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
Jan 10, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
Jan 18, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
Jan 19, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
Jan 31, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
Feb 3, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
Mar 6, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
Mar 9, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
Mar 13, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
Mar 15, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
Mar 18, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
Apr 5, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
Apr 18, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
Apr 21, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
Apr 23, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
Apr 24, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
Apr 25, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
Apr 27, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
Apr 28, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
May 1, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
May 4, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
May 6, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
May 8, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
May 13, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache9 pushed a commit
that referenced
this pull request
May 15, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/HBASE-27217