Skip to content

HBASE-22923 min version of RegionServer to move system table regions - #3438

Merged
virajjasani merged 5 commits into
apache:branch-1from
virajjasani:HBASE-22923-branch-1
Jul 1, 2021
Merged

HBASE-22923 min version of RegionServer to move system table regions#3438
virajjasani merged 5 commits into
apache:branch-1from
virajjasani:HBASE-22923-branch-1

Conversation

@virajjasani

Copy link
Copy Markdown
Contributor

No description provided.

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec11m 32sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-0 ⚠️test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ branch-1 Compile Tests _
+1 💚mvninstall9m 40sbranch-1 passed
+1 💚compile0m 40sbranch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚compile0m 44sbranch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚checkstyle1m 49sbranch-1 passed
+1 💚shadedjars3m 5sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 49sbranch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚javadoc0m 41sbranch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+0 🆗spotbugs3m 5sUsed deprecated FindBugs config; considering switching to SpotBugs.
+1 💚findbugs3m 2sbranch-1 passed
_ Patch Compile Tests _
+1 💚mvninstall1m 50sthe patch passed
+1 💚compile0m 41sthe patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚javac0m 41sthe patch passed
+1 💚compile0m 44sthe patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚javac0m 44sthe patch passed
-1 ❌checkstyle1m 37shbase-server: The patch generated 4 new + 228 unchanged - 0 fixed = 232 total (was 228)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚shadedjars2m 50spatch has no errors when building our shaded downstream artifacts.
+1 💚hadoopcheck4m 26sPatch does not cause any errors with Hadoop 2.8.5 2.9.2.
+1 💚javadoc0m 32sthe patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚javadoc0m 41sthe patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚findbugs2m 55sthe patch passed
_ Other Tests _
-1 ❌unit174m 49shbase-server in the patch failed.
+1 💚asflicense0m 40sThe patch does not generate ASF License warnings.
227m 24s
ReasonTests
Failed junit testshadoop.hbase.mapreduce.TestLoadIncrementalHFiles
hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint
hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/1/artifact/out/Dockerfile
GITHUB PR#3438
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 629130e8e8bc 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-3438/out/precommit/personality/provided.sh
git revisionbranch-1 / 395eb0c
Default JavaAzul Systems, Inc.-1.7.0_272-b10
Multi-JDK versions/usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10
checkstylehttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/1/artifact/out/diff-checkstyle-hbase-server.txt
unithttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/1/artifact/out/patch-unit-hbase-server.txt
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/1/testReport/
Max. process+thread count4214 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/1/console
versionsgit=1.9.1 maven=3.0.5 findbugs=3.0.1
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

* "hbase.min.version.move.system.tables" as "2.0.0".
* When operator uses this config, it should be used with care, meaning
* we should be confident that even if user table regions come to RS with
* higher version (that rest of cluster), it would not cause any

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The language here is ambiguous. Better to say something like

When the operator uses this configuration option, any version between the current version and the new value of "hbase.min.version.move.system.tables" does not trigger any region movement. It is assumed the configured range of versions do not require special handling.

This should also be committed to all branches, not just branch-1, for consistent functionality across all future releasing versions.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@apurtell Created master branch PR #3439 yesterday. Will create branch-2 PR as well.

*
* @return List of Excluded servers for System table regions.
*/
private List<ServerName> getExcludedServersForSystemTableUnlessAllowed() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we club the logic of both the methods and pass an optional boolean for version check?
The two functions that look very much alike except for the tail part.

@bharathv

bharathv commented Jun 29, 2021

Copy link
Copy Markdown
Contributor

Also, the "localhost" part is suspicious, why does AM look for a localhost address?

@virajjasani

Copy link
Copy Markdown
ContributorAuthor

Also, the "localhost" part is suspicious, why does AM look for a localhost address?

It is BOGUS_SERVER_NAME. AM tries to put it in FAILED_OPEN state after getting one:

 private void processBogusAssignments(Map<ServerName, List<HRegionInfo>> bulkPlan) {
if (bulkPlan.containsKey(LoadBalancer.BOGUS_SERVER_NAME)) {
// Found no plan for some regions, put those regions in RIT
for (HRegionInfo hri : bulkPlan.get(LoadBalancer.BOGUS_SERVER_NAME)) {
regionStates.updateRegionState(hri, State.FAILED_OPEN);
}
bulkPlan.remove(LoadBalancer.BOGUS_SERVER_NAME);
}
}

And it is RSGroupBasedLoadBalancer that tries to add it:

 //if not server is available assign to bogus so it ends up in RIT
if(!assignments.containsKey(LoadBalancer.BOGUS_SERVER_NAME)) {
assignments.put(LoadBalancer.BOGUS_SERVER_NAME, new ArrayList<HRegionInfo>());
}
assignments.get(LoadBalancer.BOGUS_SERVER_NAME).add(region);

and similar logic is available few other places in RSGroupBasedLoadBalancer.

Comment on lines +2598 to +2599
if (checkForMinVersion) {
if (!DEFAULT_MIN_VERSION_MOVE_SYS_TABLES_CONFIG.equals(minVersionToMoveSysTables)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

super nit: club the conditions

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the argument checkForMinVersion controls this specific flow, I thought keeping it this way would make it more readable. Thought?

@bharathvbharathvJun 30, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a strong opinion. If you ask me, I like the following version but its subjective. I'm fine with whatever you think is good.

checkForMinVersion = checkForMinVersion && !DEFAULT_MIN_VERSION_MOVE_SYS_TABLES_CONFIG.equals(minVersionToMoveSysTables);
if (checkForMinVersion) {
,,,,
}

@virajjasanivirajjasaniJul 1, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love these discussions, always learn a thing or two :)

decisionFactor = decisionFactor && additionalFactors yeah, this is also nice way. I wish we had some standards around this.
For now, let me keep it as is as you don't have strong opinion, I still find this simpler from readability viewpoint:

 if (decisionFactor) {
if (additionalFactors) {
}
}

I think this is why we don't have standards because individuals find different approaches as simpler ones than others (but CPU doesn't care :) )

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec11m 32sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-0 ⚠️test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ branch-1 Compile Tests _
+1 💚mvninstall10m 54sbranch-1 passed
+1 💚compile0m 42sbranch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚compile0m 45sbranch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚checkstyle1m 55sbranch-1 passed
+1 💚shadedjars3m 21sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 49sbranch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚javadoc0m 42sbranch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+0 🆗spotbugs3m 16sUsed deprecated FindBugs config; considering switching to SpotBugs.
+1 💚findbugs3m 12sbranch-1 passed
_ Patch Compile Tests _
+1 💚mvninstall2m 2sthe patch passed
+1 💚compile0m 42sthe patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚javac0m 42sthe patch passed
+1 💚compile0m 47sthe patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚javac0m 47sthe patch passed
-1 ❌checkstyle1m 48shbase-server: The patch generated 4 new + 231 unchanged - 0 fixed = 235 total (was 231)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚shadedjars3m 9spatch has no errors when building our shaded downstream artifacts.
+1 💚hadoopcheck5m 0sPatch does not cause any errors with Hadoop 2.8.5 2.9.2.
+1 💚javadoc0m 32sthe patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚javadoc0m 43sthe patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚findbugs3m 9sthe patch passed
_ Other Tests _
-1 ❌unit155m 4shbase-server in the patch failed.
+1 💚asflicense0m 32sThe patch does not generate ASF License warnings.
210m 58s
ReasonTests
Failed junit testshadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/2/artifact/out/Dockerfile
GITHUB PR#3438
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux eddc05d43a0b 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-agent/workspace/Base-PreCommit-GitHub-PR_PR-3438/out/precommit/personality/provided.sh
git revisionbranch-1 / 28f36f4
Default JavaAzul Systems, Inc.-1.7.0_272-b10
Multi-JDK versions/usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10
checkstylehttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/2/artifact/out/diff-checkstyle-hbase-server.txt
unithttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/2/artifact/out/patch-unit-hbase-server.txt
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/2/testReport/
Max. process+thread count5023 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/2/console
versionsgit=1.9.1 maven=3.0.5 findbugs=3.0.1
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec15m 10sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-0 ⚠️test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ branch-1 Compile Tests _
+1 💚mvninstall10m 23sbranch-1 passed
+1 💚compile0m 52sbranch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚compile0m 52sbranch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚checkstyle2m 3sbranch-1 passed
+1 💚shadedjars3m 24sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 48sbranch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚javadoc0m 43sbranch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+0 🆗spotbugs3m 16sUsed deprecated FindBugs config; considering switching to SpotBugs.
+1 💚findbugs3m 12sbranch-1 passed
_ Patch Compile Tests _
+1 💚mvninstall2m 2sthe patch passed
+1 💚compile0m 41sthe patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚javac0m 41sthe patch passed
+1 💚compile0m 47sthe patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚javac0m 47sthe patch passed
-1 ❌checkstyle1m 50shbase-server: The patch generated 4 new + 231 unchanged - 0 fixed = 235 total (was 231)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚shadedjars3m 6spatch has no errors when building our shaded downstream artifacts.
+1 💚hadoopcheck4m 46sPatch does not cause any errors with Hadoop 2.8.5 2.9.2.
+1 💚javadoc0m 32sthe patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚javadoc0m 43sthe patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚findbugs3m 7sthe patch passed
_ Other Tests _
-1 ❌unit145m 55shbase-server in the patch failed.
+1 💚asflicense0m 30sThe patch does not generate ASF License warnings.
205m 16s
ReasonTests
Failed junit testshadoop.hbase.mapreduce.TestLoadIncrementalHFiles
hadoop.hbase.security.visibility.TestVisibilityLabelsWithACL
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/3/artifact/out/Dockerfile
GITHUB PR#3438
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 774aa1b1dcb2 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-3438/out/precommit/personality/provided.sh
git revisionbranch-1 / 28f36f4
Default JavaAzul Systems, Inc.-1.7.0_272-b10
Multi-JDK versions/usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10
checkstylehttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/3/artifact/out/diff-checkstyle-hbase-server.txt
unithttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/3/artifact/out/patch-unit-hbase-server.txt
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/3/testReport/
Max. process+thread count4488 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/3/console
versionsgit=1.9.1 maven=3.0.5 findbugs=3.0.1
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

virajjasani added a commit that referenced this pull request Jul 1, 2021
…3439) (#3438)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
@virajjasani
virajjasani merged commit 7d0a72b into apache:branch-1Jul 1, 2021
@virajjasani
virajjasani deleted the HBASE-22923-branch-1 branch July 1, 2021 10:51
virajjasani added a commit that referenced this pull request Jul 1, 2021
…3439) (#3438)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
virajjasani added a commit that referenced this pull request Jul 1, 2021
…3439) (#3438)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
virajjasani added a commit that referenced this pull request Jul 1, 2021
…3439) (#3438)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@virajjasani@Apache-HBase@bharathv@apurtell