Uh oh!
There was an error while loading. Please reload this page.
HBASE-23305: Master based registry implementation - #954
Conversation
bharathv
left a comment
There was a problem hiding this comment.
@apurtell@ndimiduk@saintstack Next in the series of patches that implements a master registry.
This patch is a little more involved than the earlier ones. Happy to do anything that makes it easier for you to review. Let me know.
I'm still thinking of ways to improve the test coverage, especially injecting faults and timeouts. Even without that, I think the patch is ready for review. Any feedback welcome.
Apache-HBase
commented
Dec 20, 2019
💔 -1 overall
This message was automatically generated. |
saintstack
left a comment
There was a problem hiding this comment.
Patch looks good. Need to add delay for hedged reading but that can come in follow-on.
We are definetly saying goodbye to Master being able to sit at the back of the class being able to check in just once-in-a-while... and even being able to check out for long stretches at a time. It is now critical inline factor in overall uptime. In exchange we get simplified cluster deploy and tighter control of the interface we present clients.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
| // Configured list of masters to probe the meta information from. | ||
| private final List<ServerName> masterServers; | ||
| // Controls the fan out of the hedged requests. Requests are made in batches of this number until | ||
| // all the servers are exhausted. The first returned result is passed back to the client. |
There was a problem hiding this comment.
Once we have a successful answer, do we kill/interrupt the other ongoing queries? Wondering because 100k clients going against 3 or 5 Masters will be a bit of a load.
There was a problem hiding this comment.
If N connections to the cluster, how many MasterRegistries? As many as there are Connections? Or is there just a single instance per JVM and it is shared across?
There was a problem hiding this comment.
In my earlier review I suggest we do fan out adaptively by default. If single requests are performing adequately, fanout is unnecessary load for no reward.
There was a problem hiding this comment.
Once we have a successful answer, do we kill/interrupt the other ongoing queries? Wondering because 100k clients going against 3 or 5 Masters will be a bit of a load.
We interrupt the threads. I clarified it in the new design and wrote a detailed javadoc. Let me know if it is not clear.
If N connections to the cluster, how many MasterRegistries? As many as there are Connections? Or is there just a single instance per JVM and it is shared across?
This is actually a very good point. It is not once per JVM (although, I think that makes more sense to me). Infact it is multiple instances per connection (look at the callers of AsyncRegistryFactory#getRegistry()). I think there is definitely a scope for cleanup here. Can i revisit this as a follow up?
In my earlier review I suggest we do fan out adaptively by default. If single requests are performing adequately, fanout is unnecessary load for no reward.
Totally agree. I think the hedging policy needs to be smart. I clarified this in the new patch set. Can I implement this in a follow up patch? This is already too big.
There was a problem hiding this comment.
Infact it is multiple instances per connection (look at the callers of AsyncRegistryFactory#getRegistry()). I think there is definitely a scope for cleanup here. Can i revisit this as a follow up?
This is the state of things before you arrived, right? I'd say file this as a separate ticket, as another cleanup project for rebase, or for after this code lands. I think I'd prefer to see the former, but it's not a strong preference ATM. I do consider it a blocker for back port to a branch-2, however.
| // RPC client used to talk to the masters. | ||
| private final RpcClient rpcClient; | ||
| private final RpcControllerFactory rpcControllerFactory; | ||
| private final int rpcTimeoutNs; |
There was a problem hiding this comment.
How does this timeout relate to general client rpc timeout? If I set rpc timeout for client of 10 seconds, is this timeout subsumed by the general client timeout or does this run indepentent of whatever the overall client timeout setting is?
There was a problem hiding this comment.
Cleaned this up. Does not exist any more.
| // at doRPCs(). | ||
| @VisibleForTesting | ||
| @FunctionalInterface | ||
| public interface RpcCall<RESP> { |
There was a problem hiding this comment.
This is like AsyncAdminRequestRetryingCaller#Callable only it throws a ServiceException. Should it take a Controller? Should it be in a class of its own?
On ClientMetaService, 'meta' is overloaded in hbase. Usually it is about the hbase:meta table but here it is about something else? Should it be called something other than ClientMetaService? Somethign to do w/ Registry?
There was a problem hiding this comment.
Should it return CompleteableFuture since we in async land?
There was a problem hiding this comment.
Removed this and implemented a generic hedging framework for any RPC at the rpc layer.
| */ | ||
| private void parseMasterAddrs(Configuration conf) { | ||
| String configuredMasters = conf.get(MASTER_ADDRS_KEY, MASTER_ADDRS_DEFAULT); | ||
| for (String masterAddr: configuredMasters.split(",")) { |
There was a problem hiding this comment.
nit: Fragile? If space after comma, this breaks?
There was a problem hiding this comment.
Yikes. Added some test coverage too.
| masterServers.add(ServerName.valueOf(masterAddr, ServerName.NON_STARTCODE)); | ||
| } | ||
| // (Pseudo) Randomized so that not all clients hot spot the same set of masters. | ||
| Collections.shuffle(masterServers); |
| conf.getInt(MASTER_REGISTRY_NUM_HEDGED_REQS_KEY, MASTER_REGISTRY_NUM_HEDGED_REQS_DEFAULT); | ||
| Preconditions.checkArgument(requestFanOut >= 1); | ||
| if (requestFanOut > 1) { | ||
| masterRpcPool = Executors.newFixedThreadPool(requestFanOut, |
There was a problem hiding this comment.
TODO: delay query of second and third masters?
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
| public class MasterRegistryFetchException extends HBaseIOException { | ||
| public MasterRegistryFetchException(List<ServerName> masters, String failedRPC) { | ||
| super(String.format("Exception making rpc %s to masters %s", failedRPC, | ||
| masters.stream().map(Objects::toString).collect(Collectors.toList()))); |
There was a problem hiding this comment.
Much of this work has these, all presenting issues for backport (not insurmountable, though), but this one is less readable than a simple string join using Collections.
There was a problem hiding this comment.
Moved this to PrettyPrinter as a general util for any collection.
but this one is less readable than a simple string join using Collections.
Fair point. Switched to good old better performing loops.
Uh oh!
There was an error while loading. Please reload this page.
| @VisibleForTesting | ||
| <RESP> Optional<RESP> doRPCs(RpcCall<RESP> rpcCall, | ||
| Function<RESP, Boolean> isvalidResp, String debug) { | ||
| if (requestFanOut == 1) { |
There was a problem hiding this comment.
I was expecting for the first cut we'd do just one request to a random host on the list at a time, and retry with another random choice. (So above randomization of list is good and important.) This is what the zookeeper client does now so is no different from current state of play.
Hedged reading is ahead of the game.
Good that it is off by default, though.
Also, it's nice that fan out factor is configurable, but I would want an adaptive policy by default. Only if single requests are failing at some threshold of unacceptable probability (i suppose controlled by a config param) would you want to start loading up more than one per request in trade for faster response, hopefully, on average.
There was a problem hiding this comment.
Let's generalize this and apply it to and reuse the existing RPC controller, RPC client, Callable, Call, Caller hierarchy.
There was a problem hiding this comment.
Done. Re-did the whole thing. Now the hedging happens in the RPC layer and is not specific to MasterRegistry anymore.
| Preconditions.checkState(requestFanOut > 1); | ||
| Preconditions.checkNotNull(masterRpcPool); | ||
| int i = 0; | ||
| while (i < masterServers.size()){ |
There was a problem hiding this comment.
See above comment regarding adaptive behavior. I'd prefer if we have fan out, to not blindly do it if single requests are performing adequately.
| while (i < masterServers.size()){ | ||
| // Each iteration of loop picks requestFanOut masters | ||
| int subListSize = Math.min(masterServers.size(), i + requestFanOut); | ||
| List<ServerName> masterSubList = masterServers.subList(i, subListSize); |
There was a problem hiding this comment.
Shouldn't the list be randomized again? Or we're hitting the sublists deterministically. Make a private list at top of function and shuffle?
There was a problem hiding this comment.
I was thinking it'd be non-deterministic at the registry level. (different connections have different registries anyway, so wondering if we need to shuffle again.
| private final Function<RESP, Boolean> isValidResp; | ||
| private final String debugStr; | ||
| MasterRpc(BatchRpcCtx<RESP> rpcCtx, ServerName master, RpcCall<RESP> rpcCall, |
There was a problem hiding this comment.
We don't need a new one, right? Use existing RPC/Call facilities and set a retry policy where there are no retries using the factory methods for that.
Apache-HBase
commented
Dec 21, 2019
💔 -1 overall
This message was automatically generated. |
bharathv
left a comment
There was a problem hiding this comment.
Re-implemented the whole hedging logic again. Will force-push the new set of commits shortly.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
| // Configured list of masters to probe the meta information from. | ||
| private final List<ServerName> masterServers; | ||
| // Controls the fan out of the hedged requests. Requests are made in batches of this number until | ||
| // all the servers are exhausted. The first returned result is passed back to the client. |
There was a problem hiding this comment.
Once we have a successful answer, do we kill/interrupt the other ongoing queries? Wondering because 100k clients going against 3 or 5 Masters will be a bit of a load.
We interrupt the threads. I clarified it in the new design and wrote a detailed javadoc. Let me know if it is not clear.
If N connections to the cluster, how many MasterRegistries? As many as there are Connections? Or is there just a single instance per JVM and it is shared across?
This is actually a very good point. It is not once per JVM (although, I think that makes more sense to me). Infact it is multiple instances per connection (look at the callers of AsyncRegistryFactory#getRegistry()). I think there is definitely a scope for cleanup here. Can i revisit this as a follow up?
In my earlier review I suggest we do fan out adaptively by default. If single requests are performing adequately, fanout is unnecessary load for no reward.
Totally agree. I think the hedging policy needs to be smart. I clarified this in the new patch set. Can I implement this in a follow up patch? This is already too big.
| */ | ||
| private void parseMasterAddrs(Configuration conf) { | ||
| String configuredMasters = conf.get(MASTER_ADDRS_KEY, MASTER_ADDRS_DEFAULT); | ||
| for (String masterAddr: configuredMasters.split(",")) { |
There was a problem hiding this comment.
Yikes. Added some test coverage too.
| @VisibleForTesting | ||
| <RESP> Optional<RESP> doRPCs(RpcCall<RESP> rpcCall, | ||
| Function<RESP, Boolean> isvalidResp, String debug) { | ||
| if (requestFanOut == 1) { |
There was a problem hiding this comment.
Done. Re-did the whole thing. Now the hedging happens in the RPC layer and is not specific to MasterRegistry anymore.
| Preconditions.checkState(requestFanOut > 1); | ||
| Preconditions.checkNotNull(masterRpcPool); | ||
| int i = 0; | ||
| while (i < masterServers.size()){ |
| while (i < masterServers.size()){ | ||
| // Each iteration of loop picks requestFanOut masters | ||
| int subListSize = Math.min(masterServers.size(), i + requestFanOut); | ||
| List<ServerName> masterSubList = masterServers.subList(i, subListSize); |
There was a problem hiding this comment.
I was thinking it'd be non-deterministic at the registry level. (different connections have different registries anyway, so wondering if we need to shuffle again.
| public class MasterRegistryFetchException extends HBaseIOException { | ||
| public MasterRegistryFetchException(List<ServerName> masters, String failedRPC) { | ||
| super(String.format("Exception making rpc %s to masters %s", failedRPC, | ||
| masters.stream().map(Objects::toString).collect(Collectors.toList()))); |
There was a problem hiding this comment.
Moved this to PrettyPrinter as a general util for any collection.
but this one is less readable than a simple string join using Collections.
Fair point. Switched to good old better performing loops.
bharathv
left a comment
There was a problem hiding this comment.
To summarize the set of changes in the latest force fush.
- Hedging now happens in the rpc layer and is generic for any call.
- Implemented MasterRegistry using this hedging rpc framework.
- Clarified the hedging design in the javadocs.
- Does not yet implement the adaptable hedging logic, since the patch is already too big. I think it can be done as a follow up.
Apache-HBase
commented
Dec 25, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 25, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 26, 2019
💔 -1 overall
This message was automatically generated. |
bharathv
commented
Dec 26, 2019
Fixed all the test failures from the last run in the latest push. One of the fixes comes in as a separate PR to master #963 . To pull that in, the current feature branch needs a rebase on master. Also, from the last test run, the thread usage has substantially dropped due to the test cleanup. I think this is in a good shape for 2nd round of review. Appreciate any feedback. |
e41b46c to
1c41b36Comparebharathv
commented
Dec 31, 2019
@saintstack / @apurtell Any thoughts on the new design? Happy to address any concerns. Thanks. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
| /** | ||
| * Exception thrown when an master registry RPC fails in client. The exception includes the list of | ||
| * masters to which RPC was attempted. |
There was a problem hiding this comment.
Should this include a throwable associated with each master? They can be failing for different reasons.
There was a problem hiding this comment.
They will be logged anyway, didn't want to include them here because it will be too verbose. Thoughts?
| if (conn.getLastTouched() < closeBeforeTime && !conn.isActive()) { | ||
| if (LOG.isTraceEnabled()) LOG.trace("Cleanup idle connection to " + conn.remoteId().address); | ||
| if (LOG.isTraceEnabled()) { | ||
| LOG.trace("Cleanup idle connection to " + conn.remoteId().address); |
| // Configured list of masters to probe the meta information from. | ||
| private final List<ServerName> masterServers; | ||
| // Controls the fan out of the hedged requests. Requests are made in batches of this number until | ||
| // all the servers are exhausted. The first returned result is passed back to the client. |
There was a problem hiding this comment.
Infact it is multiple instances per connection (look at the callers of AsyncRegistryFactory#getRegistry()). I think there is definitely a scope for cleanup here. Can i revisit this as a follow up?
This is the state of things before you arrived, right? I'd say file this as a separate ticket, as another cleanup project for rebase, or for after this code lands. I think I'd prefer to see the former, but it's not a strong preference ATM. I do consider it a blocker for back port to a branch-2, however.
Uh oh!
There was an error while loading. Please reload this page.
| class HedgedRpcChannel implements RpcChannel { | ||
| private static final Logger LOG = LoggerFactory.getLogger(HedgedRpcChannel.class); | ||
| private final AbstractRpcClient rpcClient; |
There was a problem hiding this comment.
This instance variable needs to be templatized. Probably <?> is sufficient.
| * @param result Result to be set. | ||
| */ | ||
| public void setResultIfNotSet(Message result, HBaseRpcController rpcController) { | ||
| if (result == null) { |
There was a problem hiding this comment.
Odd that the null check is vs result but it's the value of rpcController.getFailed() that is actually used. I would expect the null check against the value of getFailed. There would need to be a subsequent null check against result to check for a should-not-happen kind of scenario.
| // Cancel all pending in flight calls. | ||
| for (Call call: callsInFlight) { | ||
| // It is ok to do it for all calls as it is a no-op if the call is already done. | ||
| call.setException(new CallCancelledException("Hedged call succeeded.")); |
There was a problem hiding this comment.
Perhaps something more descriptive like "Canceled because sibling hedged call succeeded". It's odd to see the message explaining an exception describe a successful result.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
| public void testHedgedAsyncTimeouts() throws Exception { | ||
| List<RpcServer> rpcServers = new ArrayList<>(); | ||
| List<InetSocketAddress> addresses = new ArrayList<>(); | ||
| // Create a mix of running and failing servers. |
There was a problem hiding this comment.
stale comment? I only see running servers in this test.
| public static final class HMasterForTest extends HMaster { | ||
| public HMasterForTest(Configuration conf) throws IOException, KeeperException { | ||
| public HMasterForTest(Configuration conf) throws IOException { |
| } | ||
| @Test | ||
| public void testMetaRegionState() throws Exception { |
1c41b36 to
dffa9beComparendimiduk
commented
Jan 3, 2020
FYI, I rebased the feature branch onto the tip of master. |
bharathv
left a comment
There was a problem hiding this comment.
@ndimiduk Thanks for taking your time to review this. You see a ton of already-reviewed-changes here because of the force pushes to the feature branch. You shouldn't be seeing them once I rebase the patch. Will address your comments after the rebase.
Apache-HBase
commented
Jan 3, 2020
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jan 4, 2020
💔 -1 overall
This message was automatically generated. |
bharathv
commented
Jan 4, 2020
Fixed the check style issues and rebased on the latest tip of the feature branch (and squashed all the commits). Test failures seem to be flakes and run fine for me locally. I think this is good to go for another round of review. |
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org> (cherry picked from commit 62da419)
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org> (cherry picked from commit 62da419)
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org> (cherry picked from commit 62da419)
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org> (cherry picked from commit 62da419)
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org> (cherry picked from commit 62da419)
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients.
registry combinations.