Support for SetWatches command, as well as watch restoration on ZK session reconnect

[brianv0]

I've hacked in SetWatches request into a local version of Kazoo for testing.

I'm not sure if this would be a feature that's relatively useful for people to expose as an API, but I think it could be useful in the case where a connection is dropped and removed from the server before a session expires, as can be the case in client restart.

Example: Client exits abnormally. The original connection is loss, the ZooKeeper server removes the connection, which removes all watches associated with that connection (as the ServerCnxn object itself is the watcher on the server side).

User had persisted KazooClient.client_id, as well as information about paths that were being watched, and KazooClient.last_zxid.

Upon reconnect, user supplies that information back to KazooClient.
User reconstructs watcher(s).

KazooClient starts, reconnecting with the previous session ID, restoring the previous session, and immediately sets watches.

If anything has changed since the relativeZxid, those events are immediately triggered (which is why client-side watch handlers should be to be set before this request).

Here is an example of the SetWatches for serialization.py:

class SetWatches(namedtuple('SetWatches', 'relativeZxid dataWatches existsWatches childWatches')):
    type = 101

    def serialize(self):
        b = bytearray()
        b.extend(long_struct.pack(self.relativeZxid))
        self.svector(b, self.dataWatches)
        self.svector(b, self.existsWatches)
        self.svector(b, self.childWatches)
        return b

    def svector(self, b, arr):
        b.extend(int_struct.pack(len(arr)))
        for elem in arr:
            b.extend(write_string(elem))

    # Not sure what we need for deserialization here

[bbangert]

Yup, I've also been thinking about this issue as well. It was on my list of things to add, and I definitely want it in by 2.2.

[rgs1]

fwiw, i have a branch with this https://github.com/rgs1/kazoo/tree/support-setwatches. i'll rebase it on master and add some tests

[bbangert]

That would be fantastic!