Allow KeepAliveRequest to manage multiple sessions

[kuujo]

In ONOS, the Copycat client's sequencing and the variation of queries/commands and consistent levels seems to significantly limit performance across primitives. Ideally, each primitive should be able to have a separate logical client that does sequencing independently of other clients. Currently, though, only one CopycatClient can be used for each session registered with the cluster, and the overhead of keeping a session alive makes it impractical to use multiple clients on a single node.

Multiple Copycat clients on the same node should be able to share a session manager and connections while performing request/response/event sequencing independent of other clients.

[kuujo]

To be clear, the reason one primitive can slow down other primitives when sharing a client is because a read-heavy primitive that uses SEQUENTIAL read consistency can block command/query responses from a leader in sequencing until a sequential response is received, which often has to wait for entries from the leader since that client has seen recent indexes in other primitives.

[kuujo]

I have been looking in to the idea a bit more tonight, and I think it could be hugely beneficial for both stability and performance. Essentially, this idea evolved out of investigating the thread model in ONOS. What I realized was that the same arguments I made about the threading model in ONOS should apply inside the Copycat client as well. Namely, that we can only guarantee order within a single thread, so the Copycat client is imposing unnecessary overhead on one thread when it has to wait for sequencing for operations that occur another thread.

So, what I'm interested in doing is creating a thread-aware client where it treats each thread as a separate logical session. A logical session adds no overhead - all sessions within a client are still managed with a single register request and keep-alives are shared - but request/response/event sequencing occurs independently of all other sessions, and each logical session appears as a separate session to state machines as well. Each primitive would then use the new thread model to use a specific logical session.

All this would really do is extend ONOS's primitive thread model into Copycat all the way down to the transport layer. That means it would take much better advantage of Netty's concurrency. But more importantly for fault tolerance, it would isolate failures in one thread/primitive from failures in other threads/primitives. For example, if a map put command in one thread gets lost and has to be resent in order to fix sequencing in the cluster, that lost command won't affect/block e.g. a leader election command in another thread until resequencing is complete. It also means sequencing of responses and events will happen independently, so a primitive that's waiting for an event from a follower doesn't block a response from a leader. From what I've seen in the logs, I think this will have a huge impact when the cluster is under high load.