Zookeeper in 3 node Kubernetes cluster does not pass heath check in 2.8.0

[andrekramer1]

Describe the bug
Running the Kubernetes Helm chart with a 2.8.0 pulsar image, Zookeeper pod 0 runs but fails repeatedly. The rest of the Pulsar cluster is waiting on Zookeeper initialization. The Zookeeper health check is not passing "echo ruok | nc localhost 2181" is not returning "imok" - just hanging I think. So the kubernetes health check is failing repeatedly. Zookeeper is also complaining about not resolving zookeeper-1 and zookeeper-2 as usual but the zookeeper-0 log have a new NullReferenceException that we've not seen before:

10:58:55.990 [epollEventLoopGroup-4-2] WARN org.apache.zookeeper.server.NettyServerCnxnFactory - Exception caught
java.lang.NullPointerException: null
at org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelActive(NettyServerCnxnFactory.java:258) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:230) [io.netty-netty-transport-4.1.63.Final.jar:4.1.63.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:216) [io.netty-netty-transport-4.1.63.Final.jar:4.1.63.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:209) [io.netty-netty-transport-4.1.63.Final.jar:4.1.63.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1398) [io.netty-netty-transport-4.1.63.Final.jar:4.1.63.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:230) [io.netty-netty-transport-4.1.63.Final.jar:4.1.63.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:216) [io.netty-netty-transport-4.1.63.Final.jar:4.1.63.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:895) [io.netty-netty-transport-4.1.63.Final.jar:4.1.63.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:522) [io.netty-netty-transport-4.1.63.Final.jar:4.1.63.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) [io.netty-netty-transport-4.1.63.Final.jar:4.1.63.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) [io.netty-netty-transport-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty-netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [io.netty-netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) [io.netty-netty-transport-native-epoll-4.1.63.Final-linux-x86_64.jar:4.1.63.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty-netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.63.Final.jar:4.1.63.Final]
at java.lang.Thread.run(Thread.java:829) [?:?

To Reproduce
Pulsar 2.8.0 docker image on Kubernetes as deployed by Helm chart. Changing just the zookeeper image to 2.7.0 and the cluster came up.

Additional context

Possibly java 11 is part of the problem or just a bad version of Zookeeper?

[mnit016]

I've faced the same issue here when upgrade to 2.8, is any update?

[andrekramer1]

A work around is using 2.7.x image for just Zookeeper.

[jmhublar]

Yeah, this is not working at all.

[eolivelli]

@hangc0276 I believe that we should fix this problem before cutting 2.8.1

[eolivelli]

the NPE is here
https://github.com/apache/zookeeper/blob/00071dd3532f47c16e0973cc5af03cccec7ba9ab/zookeeper-server/src/main/java/org/apache/zookeeper/server/NettyServerCnxnFactory.java#L258

probably:

• there is another error in the logs
• the server is taking much time to come up

[hangc0276]

Thanks for @eolivelli 's help, from the exception stack, i looks like something wrong in zookeeper or the helm chart. Would you please paste detail logs of zookeeper? @andrekramer1 @jmhublar @mnit016

[eolivelli]

which helm chart are you using ?

the "official" Apache Pulsar Helm chart was never updated to Pulsar 2.8.0
see this PR that has not been merged
apache/pulsar-helm-chart#130

[eolivelli]

I am testing it now with my helm chart....stay tuned

[eolivelli]

@andrekramer1 this line seems relevant for upgrading ZK and Pulsar 2.8

https://github.com/apache/pulsar-helm-chart/pull/130/files#diff-7b27cbfd585af4cb29a517c905cf423c776a5a0d376940b1397e81b8daffd2b7R42

do you have TLS enabled ?

[eolivelli]

I have run my tests and with this helm chart ZooKeeper works well with 3 pods
https://github.com/datastax/pulsar-helm-chart