Skip to content

HBASE-26157 Expose some IA.LimitedPrivate interface in TestingHBaseCl… - #3643

Merged
Apache9 merged 2 commits into
apache:masterfrom
Apache9:HBASE-26157
Sep 2, 2021
Merged

HBASE-26157 Expose some IA.LimitedPrivate interface in TestingHBaseCl…#3643
Apache9 merged 2 commits into
apache:masterfrom
Apache9:HBASE-26157

Conversation

@Apache9

Copy link
Copy Markdown
Contributor

…uster

@Apache9Apache9 self-assigned this Aug 28, 2021
@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 28sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 18smaster passed
+1 💚compile0m 24smaster passed
+1 💚shadedjars8m 43sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 22smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 59sthe patch passed
+1 💚compile0m 23sthe patch passed
+1 💚javac0m 23sthe patch passed
+1 💚shadedjars8m 39spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 19sthe patch passed
_ Other Tests _
-1 ❌unit1m 48shbase-testing-util in the patch failed.
30m 39s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#3643
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 89b22aaf98e6 4.15.0-151-generic #157-Ubuntu SMP Fri Jul 9 23:07:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 07c0eee
Default JavaAdoptOpenJDK-1.8.0_282-b08
unithttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-testing-util.txt
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/1/testReport/
Max. process+thread count1978 (vs. ulimit of 30000)
modulesC: hbase-testing-util U: hbase-testing-util
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 18sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall5m 3smaster passed
+1 💚compile0m 23smaster passed
+1 💚shadedjars9m 8sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 20smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 50sthe patch passed
+1 💚compile0m 23sthe patch passed
+1 💚javac0m 23sthe patch passed
+1 💚shadedjars9m 7spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 18sthe patch passed
_ Other Tests _
-1 ❌unit1m 37shbase-testing-util in the patch failed.
33m 44s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#3643
Optional Testsjavac javadoc unit shadedjars compile
unameLinux be54b8a92d85 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 07c0eee
Default JavaAdoptOpenJDK-11.0.10+9
unithttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-testing-util.txt
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/1/testReport/
Max. process+thread count1836 (vs. ulimit of 30000)
modulesC: hbase-testing-util U: hbase-testing-util
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 27sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall4m 6smaster passed
+1 💚compile0m 35smaster passed
+1 💚checkstyle0m 16smaster passed
+1 💚spotbugs0m 42smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 38sthe patch passed
+1 💚compile0m 32sthe patch passed
+1 💚javac0m 32sthe patch passed
+1 💚checkstyle0m 14sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck18m 16sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚spotbugs0m 51sthe patch passed
_ Other Tests _
+1 💚asflicense0m 15sThe patch does not generate ASF License warnings.
37m 41s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#3643
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux 2ed8a2265b66 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 07c0eee
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count96 (vs. ulimit of 30000)
modulesC: hbase-testing-util U: hbase-testing-util
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/1/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 29sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 54smaster passed
+1 💚compile0m 24smaster passed
+1 💚shadedjars8m 45sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 20smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 57sthe patch passed
+1 💚compile0m 23sthe patch passed
+1 💚javac0m 23sthe patch passed
+1 💚shadedjars8m 38spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 19sthe patch passed
_ Other Tests _
+1 💚unit1m 51shbase-testing-util in the patch passed.
30m 18s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#3643
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 553579dcf4bd 4.15.0-151-generic #157-Ubuntu SMP Fri Jul 9 23:07:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 07c0eee
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/2/testReport/
Max. process+thread count2049 (vs. ulimit of 30000)
modulesC: hbase-testing-util U: hbase-testing-util
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 20sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 47smaster passed
+1 💚compile0m 23smaster passed
+1 💚shadedjars9m 10sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 19smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 54sthe patch passed
+1 💚compile0m 22sthe patch passed
+1 💚javac0m 22sthe patch passed
+1 💚shadedjars9m 12spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 18sthe patch passed
_ Other Tests _
+1 💚unit1m 35shbase-testing-util in the patch passed.
33m 31s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#3643
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 99a2aff6120f 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 07c0eee
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/2/testReport/
Max. process+thread count1910 (vs. ulimit of 30000)
modulesC: hbase-testing-util U: hbase-testing-util
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 28sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall3m 40smaster passed
+1 💚compile0m 31smaster passed
+1 💚checkstyle0m 14smaster passed
+1 💚spotbugs0m 38smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 38sthe patch passed
+1 💚compile0m 31sthe patch passed
+1 💚javac0m 31sthe patch passed
+1 💚checkstyle0m 14sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck18m 13sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚spotbugs0m 49sthe patch passed
_ Other Tests _
+1 💚asflicense0m 12sThe patch does not generate ASF License warnings.
37m 4s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#3643
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux 6ef07fd13de3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 07c0eee
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count95 (vs. ulimit of 30000)
modulesC: hbase-testing-util U: hbase-testing-util
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/2/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@gjacoby126

Copy link
Copy Markdown
Contributor

Thanks for these, @Apache9 . These will be helpful for the cell tag tests I mentioned earlier.

Another API sometimes needed when testing a coproc is access to the Coprocessor and/or CoprocessorEnvironment (or WALCoprocessor / WAL) to assert on some state.

All of these classes are LP(COPROC), but to get to them from the minicluster even today requires using IA.Private methods (HRegion.getCoprocessorHost, not present on the Region interface, which returns the IA.Private RegionCoprocessorHost. Same for WAL which is IA.Private but required to get to WALCoprocessor / WALObserver.

I'll keep looking to find other candidates for useful helper methods.

@Apache9

Copy link
Copy Markdown
ContributorAuthor

Thanks for these, @Apache9 . These will be helpful for the cell tag tests I mentioned earlier.

Another API sometimes needed when testing a coproc is access to the Coprocessor and/or CoprocessorEnvironment (or WALCoprocessor / WAL) to assert on some state.

All of these classes are LP(COPROC), but to get to them from the minicluster even today requires using IA.Private methods (HRegion.getCoprocessorHost, not present on the Region interface, which returns the IA.Private RegionCoprocessorHost. Same for WAL which is IA.Private but required to get to WALCoprocessor / WALObserver.

I'll keep looking to find other candidates for useful helper methods.

Got the mission. I will also try to see how to expose these things and propose new PRs to add support for them. Mind giving me some pointer on what is your current usage? Maybe in Phoenix?

And, if the approach here is OK, do you want to approve or I need to find others to get approval? Thanks.


@Override
public Optional<OnlineRegions> getOnlineRegionsInterface(ServerName serverName) {
return Optional.ofNullable(util.getMiniHBaseCluster().getRegionServer(serverName));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getRegionServer returns an HRegionServer...don't we need to keep going to the HRegionServer's RS coprocessor host to get the OnlineRegions?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The HRegionServers extends RegionServerServices, RegionServerServices extends MutableOnlineRegions, and finally MutableOnlineRegions extends OnlineRegions, so HRegionServer itself is 'OnlineRegions', we do not need to go to RS coprocessor to get it. The design here is not give CP implementations a way to get the Region interface, but do not want to give the CP implementations the ability to add or remove from the online regions directly as it could mess up the state.

* And also you could use the returned interface to get all regions on this region server, etc.
*/
@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.COPROC)
Optional<OnlineRegions> getOnlineRegionsInterface(ServerName serverName);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: does the "Interface" in the method name add any information? Seems like it could be "getOnlineRegions" to return an OnlineRegions. That's just like above, and in general throughout HBase, we have "getRegion" not "getRegionInterface" when we return a Region.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why I added the 'Interface' suffix is because I assumed the name 'getOnlineRegions ' is a bit confusing. It could also be explained as get the online regions, where we should return a list of Regions.
WDYT? I'm not a native English speaker so I'm fine with changing the name, if you think 'getOnlineRegions' is enough.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that because the return type is Optional(OnlineRegions), the method name getOnlineRegions is clear without the Interface, and we usually don't append Interface. But it's not a big deal either way.

try (RegionLocator locator = CONN.getRegionLocator(NAME)) {
loc = locator.getRegionLocation(info.getStartKey()).getServerName();
}
OnlineRegions onlineRegionsInterface = CLUSTER.getOnlineRegionsInterface(loc).get();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused why this isn't a ClassCastException going between HRegionServer and OnlineRegions but I see the test is passing in at least some of the test runs. Is there something interesting going on that I'm not understanding?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As said above, the HRegionServer just extends the OnlineRegions interface, so there is no problem. So actually in the CP implementation, when you get an OnlineRegions instance, it is actually a HRegionServer :)

@gjacoby126

Copy link
Copy Markdown
Contributor

@Apache9 - No objection to limiting the review on this PR to just the methods you already added. I'll approve this PR when the review on those are done, and we can do additional methods in later JIRAs / PRs.

@Apache9

Copy link
Copy Markdown
ContributorAuthor

@Apache9 - No objection to limiting the review on this PR to just the methods you already added. I'll approve this PR when the review on those are done, and we can do additional methods in later JIRAs / PRs.

Thank you @gjacoby126 . I've replied all your comments. Waiting for your response. Thanks.

@gjacoby126gjacoby126 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

* And also you could use the returned interface to get all regions on this region server, etc.
*/
@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.COPROC)
Optional<OnlineRegions> getOnlineRegionsInterface(ServerName serverName);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that because the return type is Optional(OnlineRegions), the method name getOnlineRegions is clear without the Interface, and we usually don't append Interface. But it's not a big deal either way.

@Apache9

Copy link
Copy Markdown
ContributorAuthor

Thanks @gjacoby126 ! Let me merge.

@Apache9
Apache9 merged commit f022692 into apache:masterSep 2, 2021
Apache9 added a commit that referenced this pull request Sep 2, 2021
…uster (#3643)
Signed-off-by: Geoffrey Jacoby <gjacoby@apache.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@Apache9@Apache-HBase@gjacoby126