-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HBASE-23305: Master based registry implementation #954
HBASE-23305: Master based registry implementation #954
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@apurtell @ndimiduk @saintstack Next in the series of patches that implements a master registry.
This patch is a little more involved than the earlier ones. Happy to do anything that makes it easier for you to review. Let me know.
I'm still thinking of ways to improve the test coverage, especially injecting faults and timeouts. Even without that, I think the patch is ready for review. Any feedback welcome.
💔 -1 overall
This message was automatically generated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Patch looks good. Need to add delay for hedged reading but that can come in follow-on.
We are definetly saying goodbye to Master being able to sit at the back of the class being able to check in just once-in-a-while... and even being able to check out for long stretches at a time. It is now critical inline factor in overall uptime. In exchange we get simplified cluster deploy and tighter control of the interface we present clients.
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MasterRegistry.java
Show resolved
Hide resolved
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MasterRegistry.java
Show resolved
Hide resolved
// Configured list of masters to probe the meta information from. | ||
private final List<ServerName> masterServers; | ||
// Controls the fan out of the hedged requests. Requests are made in batches of this number until | ||
// all the servers are exhausted. The first returned result is passed back to the client. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Once we have a successful answer, do we kill/interrupt the other ongoing queries? Wondering because 100k clients going against 3 or 5 Masters will be a bit of a load.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If N connections to the cluster, how many MasterRegistries? As many as there are Connections? Or is there just a single instance per JVM and it is shared across?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In my earlier review I suggest we do fan out adaptively by default. If single requests are performing adequately, fanout is unnecessary load for no reward.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Once we have a successful answer, do we kill/interrupt the other ongoing queries? Wondering because 100k clients going against 3 or 5 Masters will be a bit of a load.
We interrupt the threads. I clarified it in the new design and wrote a detailed javadoc. Let me know if it is not clear.
If N connections to the cluster, how many MasterRegistries? As many as there are Connections? Or is there just a single instance per JVM and it is shared across?
This is actually a very good point. It is not once per JVM (although, I think that makes more sense to me). Infact it is multiple instances per connection (look at the callers of AsyncRegistryFactory#getRegistry()). I think there is definitely a scope for cleanup here. Can i revisit this as a follow up?
In my earlier review I suggest we do fan out adaptively by default. If single requests are performing adequately, fanout is unnecessary load for no reward.
Totally agree. I think the hedging policy needs to be smart. I clarified this in the new patch set. Can I implement this in a follow up patch? This is already too big.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Infact it is multiple instances per connection (look at the callers of AsyncRegistryFactory#getRegistry()). I think there is definitely a scope for cleanup here. Can i revisit this as a follow up?
This is the state of things before you arrived, right? I'd say file this as a separate ticket, as another cleanup project for rebase, or for after this code lands. I think I'd prefer to see the former, but it's not a strong preference ATM. I do consider it a blocker for back port to a branch-2, however.
// RPC client used to talk to the masters. | ||
private final RpcClient rpcClient; | ||
private final RpcControllerFactory rpcControllerFactory; | ||
private final int rpcTimeoutNs; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How does this timeout relate to general client rpc timeout? If I set rpc timeout for client of 10 seconds, is this timeout subsumed by the general client timeout or does this run indepentent of whatever the overall client timeout setting is?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cleaned this up. Does not exist any more.
// at doRPCs(). | ||
@VisibleForTesting | ||
@FunctionalInterface | ||
public interface RpcCall<RESP> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is like AsyncAdminRequestRetryingCaller#Callable only it throws a ServiceException. Should it take a Controller? Should it be in a class of its own?
On ClientMetaService, 'meta' is overloaded in hbase. Usually it is about the hbase:meta table but here it is about something else? Should it be called something other than ClientMetaService? Somethign to do w/ Registry?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should it return CompleteableFuture since we in async land?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed this and implemented a generic hedging framework for any RPC at the rpc layer.
*/ | ||
private void parseMasterAddrs(Configuration conf) { | ||
String configuredMasters = conf.get(MASTER_ADDRS_KEY, MASTER_ADDRS_DEFAULT); | ||
for (String masterAddr: configuredMasters.split(",")) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Fragile? If space after comma, this breaks?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yikes. Added some test coverage too.
masterServers.add(ServerName.valueOf(masterAddr, ServerName.NON_STARTCODE)); | ||
} | ||
// (Pseudo) Randomized so that not all clients hot spot the same set of masters. | ||
Collections.shuffle(masterServers); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good
conf.getInt(MASTER_REGISTRY_NUM_HEDGED_REQS_KEY, MASTER_REGISTRY_NUM_HEDGED_REQS_DEFAULT); | ||
Preconditions.checkArgument(requestFanOut >= 1); | ||
if (requestFanOut > 1) { | ||
masterRpcPool = Executors.newFixedThreadPool(requestFanOut, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TODO: delay query of second and third masters?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MasterRegistry.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got to the point where I suggested a big refactor of the patch's approach and stopped there for now.
public class MasterRegistryFetchException extends HBaseIOException { | ||
public MasterRegistryFetchException(List<ServerName> masters, String failedRPC) { | ||
super(String.format("Exception making rpc %s to masters %s", failedRPC, | ||
masters.stream().map(Objects::toString).collect(Collectors.toList()))); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unnecessary Java 8 idioms
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Much of this work has these, all presenting issues for backport (not insurmountable, though), but this one is less readable than a simple string join using Collections.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Moved this to PrettyPrinter as a general util for any collection.
but this one is less readable than a simple string join using Collections.
Fair point. Switched to good old better performing loops.
@VisibleForTesting | ||
<RESP> Optional<RESP> doRPCs(RpcCall<RESP> rpcCall, | ||
Function<RESP, Boolean> isvalidResp, String debug) { | ||
if (requestFanOut == 1) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was expecting for the first cut we'd do just one request to a random host on the list at a time, and retry with another random choice. (So above randomization of list is good and important.) This is what the zookeeper client does now so is no different from current state of play.
Hedged reading is ahead of the game.
Good that it is off by default, though.
Also, it's nice that fan out factor is configurable, but I would want an adaptive policy by default. Only if single requests are failing at some threshold of unacceptable probability (i suppose controlled by a config param) would you want to start loading up more than one per request in trade for faster response, hopefully, on average.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's generalize this and apply it to and reuse the existing RPC controller, RPC client, Callable, Call, Caller hierarchy.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. Re-did the whole thing. Now the hedging happens in the RPC layer and is not specific to MasterRegistry anymore.
Preconditions.checkState(requestFanOut > 1); | ||
Preconditions.checkNotNull(masterRpcPool); | ||
int i = 0; | ||
while (i < masterServers.size()){ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See above comment regarding adaptive behavior. I'd prefer if we have fan out, to not blindly do it if single requests are performing adequately.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree.
while (i < masterServers.size()){ | ||
// Each iteration of loop picks requestFanOut masters | ||
int subListSize = Math.min(masterServers.size(), i + requestFanOut); | ||
List<ServerName> masterSubList = masterServers.subList(i, subListSize); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't the list be randomized again? Or we're hitting the sublists deterministically. Make a private list at top of function and shuffle?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was thinking it'd be non-deterministic at the registry level. (different connections have different registries anyway, so wondering if we need to shuffle again.
private final Function<RESP, Boolean> isValidResp; | ||
private final String debugStr; | ||
|
||
MasterRpc(BatchRpcCtx<RESP> rpcCtx, ServerName master, RpcCall<RESP> rpcCall, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't need a new one, right? Use existing RPC/Call facilities and set a retry policy where there are no retries using the factory methods for that.
💔 -1 overall
This message was automatically generated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Re-implemented the whole hedging logic again. Will force-push the new set of commits shortly.
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MasterRegistry.java
Show resolved
Hide resolved
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MasterRegistry.java
Outdated
Show resolved
Hide resolved
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRegistryFactory.java
Outdated
Show resolved
Hide resolved
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MasterRegistry.java
Show resolved
Hide resolved
// Configured list of masters to probe the meta information from. | ||
private final List<ServerName> masterServers; | ||
// Controls the fan out of the hedged requests. Requests are made in batches of this number until | ||
// all the servers are exhausted. The first returned result is passed back to the client. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Once we have a successful answer, do we kill/interrupt the other ongoing queries? Wondering because 100k clients going against 3 or 5 Masters will be a bit of a load.
We interrupt the threads. I clarified it in the new design and wrote a detailed javadoc. Let me know if it is not clear.
If N connections to the cluster, how many MasterRegistries? As many as there are Connections? Or is there just a single instance per JVM and it is shared across?
This is actually a very good point. It is not once per JVM (although, I think that makes more sense to me). Infact it is multiple instances per connection (look at the callers of AsyncRegistryFactory#getRegistry()). I think there is definitely a scope for cleanup here. Can i revisit this as a follow up?
In my earlier review I suggest we do fan out adaptively by default. If single requests are performing adequately, fanout is unnecessary load for no reward.
Totally agree. I think the hedging policy needs to be smart. I clarified this in the new patch set. Can I implement this in a follow up patch? This is already too big.
*/ | ||
private void parseMasterAddrs(Configuration conf) { | ||
String configuredMasters = conf.get(MASTER_ADDRS_KEY, MASTER_ADDRS_DEFAULT); | ||
for (String masterAddr: configuredMasters.split(",")) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yikes. Added some test coverage too.
@VisibleForTesting | ||
<RESP> Optional<RESP> doRPCs(RpcCall<RESP> rpcCall, | ||
Function<RESP, Boolean> isvalidResp, String debug) { | ||
if (requestFanOut == 1) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. Re-did the whole thing. Now the hedging happens in the RPC layer and is not specific to MasterRegistry anymore.
Preconditions.checkState(requestFanOut > 1); | ||
Preconditions.checkNotNull(masterRpcPool); | ||
int i = 0; | ||
while (i < masterServers.size()){ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree.
while (i < masterServers.size()){ | ||
// Each iteration of loop picks requestFanOut masters | ||
int subListSize = Math.min(masterServers.size(), i + requestFanOut); | ||
List<ServerName> masterSubList = masterServers.subList(i, subListSize); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was thinking it'd be non-deterministic at the registry level. (different connections have different registries anyway, so wondering if we need to shuffle again.
public class MasterRegistryFetchException extends HBaseIOException { | ||
public MasterRegistryFetchException(List<ServerName> masters, String failedRPC) { | ||
super(String.format("Exception making rpc %s to masters %s", failedRPC, | ||
masters.stream().map(Objects::toString).collect(Collectors.toList()))); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Moved this to PrettyPrinter as a general util for any collection.
but this one is less readable than a simple string join using Collections.
Fair point. Switched to good old better performing loops.
fed4495
to
da0dfb6
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To summarize the set of changes in the latest force fush.
- Hedging now happens in the rpc layer and is generic for any call.
- Implemented MasterRegistry using this hedging rpc framework.
- Clarified the hedging design in the javadocs.
- Does not yet implement the adaptable hedging logic, since the patch is already too big. I think it can be done as a follow up.
💔 -1 overall
This message was automatically generated. |
da0dfb6
to
5ba5caa
Compare
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
6b18bae
to
641a8c5
Compare
Fixed all the test failures from the last run in the latest push. One of the fixes comes in as a separate PR to master #963 . To pull that in, the current feature branch needs a rebase on master. Also, from the last test run, the thread usage has substantially dropped due to the test cleanup. I think this is in a good shape for 2nd round of review. Appreciate any feedback. |
e41b46c
to
1c41b36
Compare
@saintstack / @apurtell Any thoughts on the new design? Happy to address any concerns. Thanks. |
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MasterRegistry.java
Show resolved
Hide resolved
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MasterRegistry.java
Outdated
Show resolved
Hide resolved
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MasterRegistry.java
Outdated
Show resolved
Hide resolved
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MasterRegistry.java
Show resolved
Hide resolved
|
||
/** | ||
* Exception thrown when an master registry RPC fails in client. The exception includes the list of | ||
* masters to which RPC was attempted. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this include a throwable associated with each master? They can be failing for different reasons.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They will be logged anyway, didn't want to include them here because it will be too verbose. Thoughts?
@@ -217,7 +215,9 @@ private void cleanupIdleConnections() { | |||
// have some pending calls on connection so we should not shutdown the connection outside. | |||
// The connection itself will disconnect if there is no pending call for maxIdleTime. | |||
if (conn.getLastTouched() < closeBeforeTime && !conn.isActive()) { | |||
if (LOG.isTraceEnabled()) LOG.trace("Cleanup idle connection to " + conn.remoteId().address); | |||
if (LOG.isTraceEnabled()) { | |||
LOG.trace("Cleanup idle connection to " + conn.remoteId().address); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: use format string.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
// Configured list of masters to probe the meta information from. | ||
private final List<ServerName> masterServers; | ||
// Controls the fan out of the hedged requests. Requests are made in batches of this number until | ||
// all the servers are exhausted. The first returned result is passed back to the client. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Infact it is multiple instances per connection (look at the callers of AsyncRegistryFactory#getRegistry()). I think there is definitely a scope for cleanup here. Can i revisit this as a follow up?
This is the state of things before you arrived, right? I'd say file this as a separate ticket, as another cleanup project for rebase, or for after this code lands. I think I'd prefer to see the former, but it's not a strong preference ATM. I do consider it a blocker for back port to a branch-2, however.
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/HedgedRpcChannel.java
Show resolved
Hide resolved
class HedgedRpcChannel implements RpcChannel { | ||
private static final Logger LOG = LoggerFactory.getLogger(HedgedRpcChannel.class); | ||
|
||
private final AbstractRpcClient rpcClient; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This instance variable needs to be templatized. Probably <?>
is sufficient.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
* @param result Result to be set. | ||
*/ | ||
public void setResultIfNotSet(Message result, HBaseRpcController rpcController) { | ||
if (result == null) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Odd that the null check is vs result
but it's the value of rpcController.getFailed()
that is actually used. I would expect the null check against the value of getFailed. There would need to be a subsequent null check against result
to check for a should-not-happen kind of scenario.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense.
// Cancel all pending in flight calls. | ||
for (Call call: callsInFlight) { | ||
// It is ok to do it for all calls as it is a no-op if the call is already done. | ||
call.setException(new CallCancelledException("Hedged call succeeded.")); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps something more descriptive like "Canceled because sibling hedged call succeeded". It's odd to see the message explaining an exception describe a successful result.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/HedgedRpcChannel.java
Show resolved
Hide resolved
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestScannersFromClientSide.java
Show resolved
Hide resolved
hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/AbstractTestIPC.java
Show resolved
Hide resolved
public void testHedgedAsyncTimeouts() throws Exception { | ||
List<RpcServer> rpcServers = new ArrayList<>(); | ||
List<InetSocketAddress> addresses = new ArrayList<>(); | ||
// Create a mix of running and failing servers. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
stale comment? I only see running servers in this test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done.
@@ -53,7 +52,7 @@ | |||
|
|||
public static final class HMasterForTest extends HMaster { | |||
|
|||
public HMasterForTest(Configuration conf) throws IOException, KeeperException { | |||
public HMasterForTest(Configuration conf) throws IOException { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Your second miracle.
@@ -348,4 +352,32 @@ public void testToCell() throws Exception { | |||
ProtobufUtil.toCell(ExtendedCellBuilderFactory.create(CellBuilderType.SHALLOW_COPY), cell); | |||
assertTrue(CellComparatorImpl.COMPARATOR.compare(offheapKV, newOffheapKV) == 0); | |||
} | |||
|
|||
@Test | |||
public void testMetaRegionState() throws Exception { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
💯
1c41b36
to
dffa9be
Compare
FYI, I rebased the feature branch onto the tip of master. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ndimiduk Thanks for taking your time to review this. You see a ton of already-reviewed-changes here because of the force pushes to the feature branch. You shouldn't be seeing them once I rebase the patch. Will address your comments after the rebase.
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
Fixed the check style issues and rebased on the latest tip of the feature branch (and squashed all the commits). Test failures seem to be flakes and run fine for me locally. I think this is good to go for another round of review. |
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org> (cherry picked from commit 62da419)
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org> (cherry picked from commit 62da419)
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org> (cherry picked from commit 62da419)
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org> (cherry picked from commit 62da419)
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org> (cherry picked from commit 62da419)
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients. - Supports hedged RPCs (fan out configured via configs). - Parameterized existing client tests to run with multiple registry combinations. - Added unit-test coverage for the new registry implementation. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: stack <stack@apache.org> Signed-off-by: Andrew Purtell <apurtell@apache.org>
Implements a master based registry for clients.
registry combinations.