ZookeeperCacheTest gets stuck indefinitely

[merlimat]

I have noticed that since few days, few tests have started to get stuck and the build gets timed out after 100 minutes. It is happening very frequently and it was not happening before, so I suspect this is related to some recent changes.

It gets stuck mostly on ZookeeperCacheTest though I have also seen in other places.

https://builds.apache.org/job/pulsar-pull-request/1941/console

[INFO] Running org.apache.pulsar.zookeeper.ZookeeperCacheTest
Build timed out (after 100 minutes). Marking the build as aborted.
...

[merlimat]

Another failure, now with the test timeout:

Error Message

Method org.apache.pulsar.zookeeper.ZookeeperCacheTest.testChildrenCache() didn't finish within the time-out 10000

Stacktrace

org.testng.internal.thread.ThreadTimeoutException: Method org.apache.pulsar.zookeeper.ZookeeperCacheTest.testChildrenCache() didn't finish within the time-out 10000
	at java.util.concurrent.ThreadPoolExecutor.tryTerminate(ThreadPoolExecutor.java:713)
	at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1014)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

[merlimat]

It appears this is happening after the from OrderedExecutor to OrderedScheduler. The executor is rejecting after 1 task is already queued.

16:34:52.236 [main] ERROR org.apache.pulsar.zookeeper.ZooKeeperCache - Failed to updated zk-cache /test on zk-watch Queue at limit of 1 items
java.util.concurrent.RejectedExecutionException: Queue at limit of 1 items
	at org.apache.bookkeeper.common.util.BoundedScheduledExecutorService.checkQueue(BoundedScheduledExecutorService.java:138) ~[bookkeeper-server-shaded-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
	at org.apache.bookkeeper.common.util.BoundedScheduledExecutorService.submit(BoundedScheduledExecutorService.java:94) ~[bookkeeper-server-shaded-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
	at org.apache.bookkeeper.common.util.OrderedScheduler.submitOrdered(OrderedScheduler.java:327) ~[bookkeeper-server-shaded-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
	at org.apache.pulsar.zookeeper.ZooKeeperCache.process(ZooKeeperCache.java:135) ~[classes/:?]
	at org.apache.pulsar.zookeeper.LocalZooKeeperCache.process(LocalZooKeeperCache.java:63) ~[classes/:?]
	at org.apache.pulsar.zookeeper.ZooKeeperChildrenCache.process(ZooKeeperChildrenCache.java:98) ~[classes/:?]