Auto refresh a list of cluster nodes in TarantoolClusterClient

[hypersolid]

https://github.com/tarantool/vshard is current cluster solution for Tarantool. It introduces concept of "routers" - Tarantool instances which route requests to appropriate "storage" instances where data is actually stored.

I think we could add TarantoolClusterClient for tarantool clusters, which will round-robin requests to a group of Tarantool instances. It will be also useful to be able to pass custom sharding function to it (for custom sharding solutions).

[sharonovd]

Since we now have TarantoolClusterClient, we could add the following to it:
[ ] ability to extract list of available nodes from first connected node and add them to TarantoolClusterClientConfig
[ ] ability to refresh this list periodically

The procedure called on Tarantool side to determine the list of available nodes must be pluggable, since we have several use-cases (sharded cluster, replicaset, 'cluster' module etc.)

We may consider round robin between available instances

[Totktonada]

Notes from meeting with Dmitry (@sharonovd) (Jan, 31).

• Is it possible that the procedure will return an id that is not an URI?

  • The procedure should return list of URIs. It is okay to expect in the java connector that all URIs will be in the inet domain ([user[:pass]@]host:port), because the connector does not support unix sockets.

• Whether we should store arguments in TarantoolClusterClientConfig and pass them to the procedure?

  • It is not so needed, so let's implement it only if it is simple.

• What to do if a node we currently connected disappears from the list?

  • Raw idea: If there are no errors in last requests we can leave remaining requests to end with the current node, but close it for new request. It doesn't look as a simple solution.
  • We can retry all requests that were on the fly with the new node, but anyway we need to handle DML requests correctly (ensure we don't do it twice).

• Whether we want to support a list of instances that can be asked for cluster URLs instead of just one?

  • We need to elaborate our behaviour in the case when a node is dead when we try to fetch the list at the first time or update the list.
  • It seems that we need to implement round robin across these informational nodes too.
  • Raw idea: The round robin logic can be separated from certain actions one want to do on these nodes. It can be implemented as a separate class that holds a list of URLs, but doesn't hold a connection for one of them. It provides call method and connects to a node (try each across the list) right during a call to this method.

• NB: Look at the couchbase API.

[Totktonada]

Whether we want to support a list of instances that can be asked for cluster URLs instead of just one?

I thought about that again and decided that there are no much sense to support a list of instances here. There are three possible situations when we need to switch from one instance to an another one to obtain a list of node in a cluster, all are about unavailability of the original node: accidental node downtime, planned maintanence on the node, cluster updating with moving out the node.

Accidental node downtime will break fetching a list of nodes, but DQL/DML requests will be switched by a cluster client onto another node. The nodes list updating is not a final goal; it'll stuck for some time, but DQL/DML will work. So this case does not show us any reason to expand 'info node' to a list of such nodes.

Planned maintanence when the node is temporary down is quite same.

When we want to move the node out of a cluster it is some kind of planned work. Expanding 'info node' to a list would not save us from a situation when all nodes in the list will be down (sooner or later). So we anyway should include updating of an info node address / list into the plan for such cluster reconstruction. A list does not add any value here.

[ponomarevDmitri]

Ok, the final algorithm will consist from following steps:

1. Configuring. A connector client must specify an 'info node' address as host:port string and username/password as separate fields in a cluster config (like it've been made in current cluster config). Also the list must specify a function on a tarantool instance side that have to return a list of strings which consists from 'host:port' addresses of nodes (let's call them 'target nodes').
2. Fetching list of nodes. The connector fetches list of cluster nodes using the function from 'info node'. (This operation can run in background)
3. Updating the target nodes list.
   If newly obtained list does not contain the current node then wait responses of requests which've been sent to this node in background. Retry failed request at nodes from the new list. Instant failing the sent CUD requests make less sense cause if the node is down then all requests will be automatically retried anyways, or if the node is alive then the requests will be completed. It will be more transparent for a connector client - he will know nothing about hypothetical node fail.
4. The client round-robins amid nodes in the list.
5. Update the node list by timeout (return to step 2))

[Totktonada]

There is the question how to handle the situation when a new list of nodes
does not contain a node we currectly connected. We need to switch a cluster
client to one of the new nodes. It is possible that we'll have some amount of
requests 'in fly': they were sent, but responses are not received for now.

There are the following ways to handle this situation:

1. Pause new requests (don't send them, but add to a separate queue), wait for
   'in fly' requests, then switch node.
2. Pause switching to a new node (and send all new requests to an old node)
   until we'll detect one of two situations: there are no requests 'in fly' or
   a reconnect occurs.
3. Wait for responses to requests 'in fly', but send all new requests to a new
   node.

Third way requires to store two sockets in a cluster client and handle both.
readPacket() and other utility methods from TarantoolBase need to be
factored out to be able to work with arbitrary socket. That is why refactoring
in PR #135 was made. Maybe also other refactoring needed.

I don't sure it was right thing to inherit TarantoolClusterClient from
TarantoolClientImpl (we add round robin socket provider and retry strategy to
a regular client in a cluster client). Maybe this would be more natural if we
would store several TarantoolClientImpl instances in TarantoolClusterClient?
Don't sure, so consider this as a raw idea.

I would say that 2nd way should be less intruisive considering our current
architerture. But it is up to a developer which way is the best one
considering amount of work (the deadline is near), complexity of changes (size
of a patch) and resulting code readability.