ChildrenWatch should not log exceptions if the node its watching is deleted

[lcosmin]

Given the following code:

import sys
import logging

logging.basicConfig(level=logging.DEBUG)

from time import sleep
from kazoo.client import KazooClient
from kazoo.recipe.watchers import ChildrenWatch

def child_watch(c):
    print "child_watch: {}".format(c)

zk = KazooClient("127.0.0.1:2181")
zk.start()

watch = ChildrenWatch(zk, "/test", child_watch)

while True:
    sleep(1)
    print ".",
    sys.stdout.flush()

How can I gracefully handle the removal of /test from ZooKeeper ? The code raises the following exception, apparently in a watcher thread and I can't act based on that.

I used a DataWatch to work around this issue and re-set the ChildrenWatch when /test reappears, but the presence of the exception makes me think there's a bug/unhandled exception case in the kazoo code.

INFO:kazoo.client:Received EVENT: Watch(type=4, state=3, path=u'/test')
INFO:kazoo.client:Sending request(xid=2): GetChildren(path='/test', watcher=<bound method ChildrenWatch._watcher of <kazoo.recipe.watchers.ChildrenWatch object at 0x102271910>>)
DEBUG:kazoo.client:Reading for header ReplyHeader(xid=2, zxid=103345, err=0)
INFO:kazoo.client:Received response(xid=2): []
 child_watch: []
. . . .DEBUG:kazoo.client:Sending request(xid=-2): Ping()
DEBUG:kazoo.client:Received Ping
 . . .DEBUG:kazoo.client:Sending request(xid=-2): Ping()
DEBUG:kazoo.client:Received Ping
 .INFO:kazoo.client:Received EVENT: Watch(type=2, state=3, path=u'/test')
INFO:kazoo.client:Sending request(xid=3): GetChildren(path='/test', watcher=<bound method ChildrenWatch._watcher of <kazoo.recipe.watchers.ChildrenWatch object at 0x102271910>>)
DEBUG:kazoo.client:Reading for header ReplyHeader(xid=3, zxid=103346, err=-101)
INFO:kazoo.client:Received error(xid=3) NoNodeError((), {})
ERROR:kazoo.handlers.threading:Exception in worker queue thread
Traceback (most recent call last):
  File "/home/user/.pyenv/versions/mac_venv/lib/python2.7/site-packages/kazoo/handlers/threading.py", line 201, in thread_worker
    func()
  File "/home/user/.pyenv/versions/mac_venv/lib/python2.7/site-packages/kazoo/handlers/threading.py", line 289, in <lambda>
    self.callback_queue.put(lambda: callback.func(*callback.args))
  File "/home/user/.pyenv/versions/mac_venv/lib/python2.7/site-packages/kazoo/recipe/watchers.py", line 305, in _watcher
    self._get_children(event)
  File "/home/user/.pyenv/versions/mac_venv/lib/python2.7/site-packages/kazoo/recipe/watchers.py", line 22, in wrapper
    return func(*args, **kwargs)
  File "/home/user/.pyenv/versions/mac_venv/lib/python2.7/site-packages/kazoo/recipe/watchers.py", line 283, in _get_children
    self._path, self._watcher)
  File "/home/user/.pyenv/versions/mac_venv/lib/python2.7/site-packages/kazoo/retry.py", line 122, in __call__
    return func(*args, **kwargs)
  File "/home/user/.pyenv/versions/mac_venv/lib/python2.7/site-packages/kazoo/client.py", line 945, in get_children
    return self.get_children_async(path, watch, include_data).get()
  File "/home/user/.pyenv/versions/mac_venv/lib/python2.7/site-packages/kazoo/handlers/threading.py", line 107, in get
    raise self._exception
NoNodeError: ((), {})
 . . . .DEBUG:kazoo.client:Sending request(xid=-2): Ping()
DEBUG:kazoo.client:Received Ping

[bbangert]

What is the desired behavior? I can update it so that it doesn't throw an exception and instead stops gracefully if the node is destroyed.

[lcosmin]

In my opinion the desired behaviour would be to allow the client code to handle the error case itself instead of the library code logging the exception. It feels weird for my application to display tracebacks which I cannot intercept/hide.

[bbangert]

Wouldn't client code do that by placing a watch on the node as you suggest to determine if its deleted? In which case the desired behavior would be for this exception to be masked, right?

[lcosmin]

Indeed, I can detect when the node gets deleted using a DataWatch, but I will still see the exception traceback. I guess I wasn't so clear in my previous reply, my problem is that I see an exception traceback in my logs for something I don't consider to be an exception scenario (and I can't even try..except to hide it. 😃 ). So yes, I believe this exception should be masked.

[diranged]

+1 here ... this is bugging me as well. Theres nothing we can do in our code to mask the error, so to our developers it looks like theres a real problem when there may not be.

[bbangert]

I've updated the name to indicate what I believe the remedy should be. Since everything is functioning properly, logging exceptions isn't very useful. I might get to this later next week, otherwise a PR providing an option to mask it would be welcome.

[petercuichen]

+1...Would this issue be solved sometime? The exception logging really annoy me.

[HyukjinKwon]

+1 Wouldn't there be any workaround until this is resolved?

[vblz]

@tonyseek #393 Also I think ChildrenWatch should stop watching, remove itself from client's listeners, etc if the node its watching is deleted.

[tonyseek]

@vblazhnov I modified the implementation to close watching if watched node gone.