Nacos Client: DefaultPublisher thread not destroyed while stop web application in tomcat

[FrioSea]

Issue Description

Nacos-client 2.2.0
Tomcat 9.0

While stop web application in tomcat, a thread called "nacos.publisher-com.alibaba.nacos.common.notify.SlowEvent" still remain in JVM. This typical thread was created by DefaultPublisher class (in DefaultPublisher file line 65).

Describe what happened (or what feature you want)

I've tried to invoke shutdown() in contextDestroy but failed to remove it. This thread seems to be blocked by a BlockingQueue.take() method:

`
void openEventHandler() {
try {

        // This variable is defined to resolve the problem which message overstock in the queue.
        int waitTimes = 60;
        // To ensure that messages are not lost, enable EventHandler when
        // waiting for the first Subscriber to register
        while (!shutdown && !hasSubscriber() && waitTimes > 0) {
            ThreadUtils.sleep(1000L);
            waitTimes--;
        }

        while (!shutdown) {
            final Event event = queue.take();
            receiveEvent(event);
            UPDATER.compareAndSet(this, lastEventSequence, Math.max(lastEventSequence, event.sequence()));
        }
    } catch (Throwable ex) {
        LOGGER.error("Event listener exception : ", ex);
    }
}`

shutdown() method would set the shutdown parameter to true, but this while loop has been blocked by queue.take(), so it won't get a chance to execute next loop.

Additionally, there are serveral ShutdownHook that been registered but never unregistered:

ThreadUtils.addShutdownHook(NotifyCenter::shutdown);
ThreadUtils.addShutdownHook(HttpClientBeanHolder::shutdown);

In tomcat, i don't think these hooks will be executed, because those workload will be took over by contextDestory. so it will remain in JVM and keep a reference to those classes and prevent them from being garbage collected, correct me if wrong.

Describe what you expected to happen

nacos thread being garbage collected after web application stopped or share those thread instead of creating new one.

How to reproduce it (as minimally and precisely as possible)

1. deploy a nacos config featured web application to Tomcat
2. stop this web application
3. open jvm trace tool to observe threads

Tell us your environment

Tomcat 9
Naocos client 2.2.0

Anything else we need to know?

1. How could i actually stop all nacos threads when stop a web application?
2. Is there a feature to not register shutdown hook when in a web environment.

[KomachiSion]

I think it can be enhance by interrupt thread or put an EmptyEvent to entry next loop after call shutdown.

[FrioSea]

I think it can be enhance by interrupt thread or put an EmptyEvent to entry next loop after call shutdown.

Thanks, I think publish an empty event after a shutdown call is a workaround, by NotifyCenter.publishEvent(new ServerlistChangeEvent(). Now this thread could be removed correctly.

And how about those two hooks? Would it be a cause that prevent classloader be gc?

[KomachiSion]

hook only can call shutdown method, but if no event read, the thread still wait to event until be interrupted or jvm close(deamon thread will be close when jvm shutdown).

[FrioSea]

hook only can call shutdown method, but if no event read, the thread still wait to event until be interrupted or jvm close(deamon thread will be close when jvm shutdown).

Yeah, sorry let me be more specific, SpringBoot with WAR deployment has disable hook registration by default , so those two hooks were registered but never unregistered, i think in web container like tomcat, each context has an independent ParallelWebClassLoader, so when a webapp get stopped JVM would not shutdown, therefore those hooks will not be executed and continuously hold a reference to some classes that is loaded by ParallelWebClassLoader and prevent this classloader from been gc. In some case after a bunch of restart or redeploy, it's very likely to cause a memoryLeak.

And this is just my brief understanding, please, correct me if wrong.

I think maybe add a state check, if hook registration is enabled? and let user end threads in servlet's contextDestroy listener.