Skip to content

HBASE-22527 [hbck2] Add a master web ui to show the problematic regions - #373

Closed
infraio wants to merge 1 commit into
apache:masterfrom
infraio:hbase-checker
Closed

HBASE-22527 [hbck2] Add a master web ui to show the problematic regions#373
infraio wants to merge 1 commit into
apache:masterfrom
infraio:hbase-checker

Conversation

@infraio

Copy link
Copy Markdown
Contributor

No description provided.

@Apache9Apache9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a UT?

import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.LinkedList;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's this...

private final RegionStates regionStates = new RegionStates();
private final RegionStateStore regionStateStore;

private final ConcurrentMap<ServerName, Set<byte[]>> rsReports = new ConcurrentHashMap<>();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like all of our access is synchronized, why are we using CHM?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to HashMap in new patch.

return master;
}

public Map<byte[], Pair<ServerName, Set<ServerName>>> getProblematicRegions() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how do we get from this method to the master UI?

we need a way to get the same information via a cli, either hbck1 or shell.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the UI in new patch.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought hbck1 already have the ability to find these problematic regions?

return master;
}

public Map<byte[], Pair<ServerName, Set<ServerName>>> getProblematicRegions() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this method needs a javadoc description of what we're returning

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the javadoc in new patch.

}

public Map<byte[], Pair<ServerName, Set<ServerName>>> getProblematicRegions() {
ConcurrentMap<byte[], Set<ServerName>> reportedOnlineRegions = new ConcurrentHashMap<>();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this a concurrent map?

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec133Docker mode activated.
_ Prechecks _
+1hbaseanti0Patch does not have any anti-patterns.
+1@author0The patch does not contain any @author tags.
-0test4tests0The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1mvninstall291master passed
+1compile62master passed
+1checkstyle82master passed
+1shadedjars285branch has no errors when building our shaded downstream artifacts.
+1findbugs268master passed
+1javadoc39master passed
_ Patch Compile Tests _
+1mvninstall276the patch passed
+1compile58the patch passed
+1javac58the patch passed
-1checkstyle78hbase-server: The patch generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9)
+1whitespace0The patch has no whitespace issues.
+1shadedjars295patch has no errors when building our shaded downstream artifacts.
+1hadoopcheck855Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1findbugs252the patch passed
+1javadoc40the patch passed
_ Other Tests _
-1unit1748hbase-server in the patch failed.
+1asflicense13The patch does not generate ASF License warnings.
5114
ReasonTests
Failed junit testshadoop.hbase.regionserver.TestMultiVersionConcurrencyControlBasic
hadoop.hbase.master.balancer.TestServerAndLoad
hadoop.hbase.regionserver.TestBlocksScanned
SubsystemReport/Notes
DockerClient=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-373/1/artifact/out/Dockerfile
GITHUB PR#373
Optional Testsdupname asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 3e020f4aae34 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 GNU/Linux
Build toolmaven
Personality/testptch/patchprocess/precommit/personality/provided.sh
git revisionmaster / 438bf32
mavenversion: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)
Default Java1.8.0_181
findbugsv3.1.11
checkstylehttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-373/1/artifact/out/diff-checkstyle-hbase-server.txt
unithttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-373/1/artifact/out/patch-unit-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-373/1/testReport/
Max. process+thread count674 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-373/1/console
Powered byApache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@infraio
infraioforce-pushed the hbase-checker branch 2 times, most recently from 7d237eb to 4b5243fCompareJuly 15, 2019 11:15
@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec159Docker mode activated.
_ Prechecks _
+1hbaseanti0Patch does not have any anti-patterns.
+1@author0The patch does not contain any @author tags.
+1test4tests0The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1mvninstall253master passed
+1compile51master passed
+1checkstyle67master passed
+1shadedjars263branch has no errors when building our shaded downstream artifacts.
+1findbugs198master passed
+1javadoc33master passed
_ Patch Compile Tests _
+1mvninstall234the patch passed
+1compile51the patch passed
+1javac51the patch passed
-1checkstyle67hbase-server: The patch generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9)
+1whitespace0The patch has no whitespace issues.
+1shadedjars263patch has no errors when building our shaded downstream artifacts.
+1hadoopcheck716Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1findbugs210the patch passed
+1javadoc32the patch passed
_ Other Tests _
-1unit13913hbase-server in the patch failed.
+1asflicense24The patch does not generate ASF License warnings.
16843
SubsystemReport/Notes
DockerClient=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-373/2/artifact/out/Dockerfile
GITHUB PR#373
Optional Testsdupname asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux ecbdac0f8b0f 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux
Build toolmaven
Personality/testptch/patchprocess/precommit/personality/provided.sh
git revisionmaster / 74731c2
mavenversion: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)
Default Java1.8.0_181
findbugsv3.1.11
checkstylehttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-373/2/artifact/out/diff-checkstyle-hbase-server.txt
unithttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-373/2/artifact/out/patch-unit-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-373/2/testReport/
Max. process+thread count4915 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-373/2/console
Powered byApache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@infraio
infraioforce-pushed the hbase-checker branch 2 times, most recently from bc945a1 to 0b6fc5dCompareJuly 16, 2019 01:42
@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec65Docker mode activated.
_ Prechecks _
+1hbaseanti0Patch does not have any anti-patterns.
+1@author0The patch does not contain any @author tags.
+1test4tests0The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1mvninstall274master passed
+1compile55master passed
+1checkstyle71master passed
+1shadedjars287branch has no errors when building our shaded downstream artifacts.
+1findbugs209master passed
+1javadoc33master passed
_ Patch Compile Tests _
+1mvninstall240the patch passed
+1compile53the patch passed
+1javac53the patch passed
-1checkstyle69hbase-server: The patch generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9)
+1whitespace0The patch has no whitespace issues.
+1shadedjars276patch has no errors when building our shaded downstream artifacts.
+1hadoopcheck834Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1findbugs206the patch passed
+1javadoc34the patch passed
_ Other Tests _
-1unit15667hbase-server in the patch failed.
+1asflicense39The patch does not generate ASF License warnings.
18716
ReasonTests
Failed junit testshadoop.hbase.client.TestAsyncRegionAdminApi2
SubsystemReport/Notes
DockerClient=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-373/3/artifact/out/Dockerfile
GITHUB PR#373
Optional Testsdupname asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 79b61896c3b9 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 GNU/Linux
Build toolmaven
Personality/testptch/patchprocess/precommit/personality/provided.sh
git revisionmaster / cc38de1
mavenversion: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)
Default Java1.8.0_181
findbugsv3.1.11
checkstylehttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-373/3/artifact/out/diff-checkstyle-hbase-server.txt
unithttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-373/3/artifact/out/patch-unit-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-373/3/testReport/
Max. process+thread count5179 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-373/3/console
Powered byApache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec28Docker mode activated.
_ Prechecks _
+1hbaseanti0Patch does not have any anti-patterns.
+1@author0The patch does not contain any @author tags.
+1test4tests0The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1mvninstall234master passed
+1compile51master passed
+1checkstyle65master passed
+1shadedjars263branch has no errors when building our shaded downstream artifacts.
+1findbugs207master passed
+1javadoc33master passed
_ Patch Compile Tests _
+1mvninstall232the patch passed
+1compile53the patch passed
+1javac53the patch passed
-1checkstyle67hbase-server: The patch generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9)
+1whitespace0The patch has no whitespace issues.
+1shadedjars268patch has no errors when building our shaded downstream artifacts.
+1hadoopcheck736Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1findbugs214the patch passed
+1javadoc32the patch passed
_ Other Tests _
+1unit7802hbase-server in the patch passed.
+1asflicense22The patch does not generate ASF License warnings.
10618
SubsystemReport/Notes
DockerClient=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-373/4/artifact/out/Dockerfile
GITHUB PR#373
Optional Testsdupname asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 8a3d2e58432a 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 GNU/Linux
Build toolmaven
Personality/testptch/patchprocess/precommit/personality/provided.sh
git revisionmaster / 72e58a8
mavenversion: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)
Default Java1.8.0_181
findbugsv3.1.11
checkstylehttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-373/4/artifact/out/diff-checkstyle-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-373/4/testReport/
Max. process+thread count4548 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-373/4/console
Powered byApache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@saintstacksaintstack left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really great. Just a note on something to add on commit.

int numOfPages = (int) Math.ceil(totalSize * 1.0 / sizePerPage);
</%java>
<section>
<h2><a name="rit">Problematic Regions</a></h2>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On commit, add a sentence that says what a problematic region is. It seems like its one that has a meta entry that does not agree w/ where it is actually deployed?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or, you say below what it is... * case 1. Master thought this region opened, but no regionserver reported it.

  • case 2. Master thought this region opened on Server1, but regionserver reported Server2
  • case 3. More than one regionservers reported opened this region

Can this be added in the UI maybe in small text?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already committed it. Will add addendum for this.

@infraioinfraio closed this Jul 18, 2019
@infraio
infraio deleted the hbase-checker branch July 18, 2019 05:53
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@infraio@Apache-HBase@busbey@saintstack@Apache9