Uh oh!
There was an error while loading. Please reload this page.
Conversation
busbey
commented
Dec 21, 2015
Contributor
Could you please open an issue on our project tracker describing the issue and attach a patch: https://issues.apache.org/jira/browse/HBASE for more information on submitting contributions to the project, please read through our new contributor section: |
busbey
commented
Mar 21, 2016
Contributor
still interested in working on this? without a response it'll get closed as stale. |
asfgit pushed a commit
that referenced
this pull request
Sep 29, 2016
… protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
asfgit pushed a commit
that referenced
this pull request
Oct 4, 2016
Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
bbeaudreault pushed a commit
that referenced
this pull request
Dec 15, 2022
HBASE-26745 Include Balancer Cost metrics in jmx endpoint
sanjeet006py pushed a commit
to sanjeet006py/hbase
that referenced
this pull request
May 14, 2026
Address point apache#4 of the canvas/code review: the paged meta scan was making ceil(limit / hbase.meta.scanner.caching) ScannerNext RPCs whenever the limit exceeded the configured caching, contradicting the "at most one RPC per invocation" javadoc on RegionLocator.getRegionLocationsPage / AsyncTableRegionLocator.getRegionLocationsPage. Plumb a private isPagedScan flag through ClientMetaTableAccessor.scanMeta and getMetaScan. When true, size both setLimit and setCaching to the caller-supplied rowUpperLimit so the slice returns in a single ScannerNext RPC regardless of the configured caching default. The unbounded scanMeta overload (used by getTableHRegionLocations(metaTable, tableName) for getAllRegionLocations) keeps its existing behavior. Add TestRegionLocatorPagedScanRpcCount: starts a mini-cluster with hbase.meta.scanner.caching=2, wraps the meta AsyncTable to count AdvancedScanResultConsumer.onNext invocations, and asserts: - limit <= caching -> 1 ScannerNext (baseline) - limit > caching -> 1 ScannerNext (the regression check) - unbounded scan -> ceil(N / caching) ScannerNext (proves the flag does not leak into the existing path) Verified the test catches the regression: reverting the getMetaScan change makes testSingleRpcWhenLimitExceedsCaching fail with onNext=3.
Umeshkumar9414 added a commit
to Umeshkumar9414/hbase
that referenced
this pull request
Aug 11, 2026
… API, Javadoc, imports - getRSGroupInfo: add LOG.debug when hasRSGroupInfo()==false to aid diagnosis of address-form mismatches (hostname vs IP when use-ip mode is active per HBASE-27304). The mismatch causes a silent fallback to all-server mode; the log makes it visible without changing behaviour (finding apache#4). - Replace deprecated Admin.getMasterCoprocessors() with getMasterCoprocessorNames() which returns List<String> directly, removing the Arrays.asList() wrapper (finding apache#5). - Fix orphaned Javadoc: the includeExcludeRegionServers doc block was stranded above getRSGroupInfo after earlier refactoring. Moved it back to its correct method; getRSGroupInfo retains its own doc block (finding apache#6). - Add imports for com.google.protobuf.ServiceException and org.apache.hadoop.hbase.protobuf.ProtobufUtil; remove inline FQNs at the catch/throw site. Also add java.util.Arrays and java.util.ArrayList imports to both test files, replacing all inline java.util.* usages (finding apache#7).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
solaris 64 fix based on https://issues.apache.org/jira/secure/attachment/12625582/solaris_unsafe_fix.patch
https://bugs.openjdk.java.net/browse/JDK-8021574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel