Consumer throws RpcException after RegistryDirectory notify in high QPS

[haiyang1985]

• I have searched the issues of this repository and believe that this is not a duplicate.
• I have checked the FAQ of this repository and believe that this is not a duplicate.

Environment

• Dubbo version: 2.5.10
• Operating System version: MacOS Sierra 10.12
• Java version: JDK 1.8

Step to reproduce this issue

1. Start dubbo provider (retries=0).
2. Start dubbo consumer*10 in 5000 QPS (provider in 5w QPS).
3. Update dynamic config(override URL) for consumer, it will trigger RegistryDirectory notify(same as DubboAdmin 动态配置).

Expected Result

The dynamic config(override url) takes effect after RegistryDirectory notify, without RpcException .

Actual Result

What is actually happen?
Sometimes, client throws RpcException after dynamic config, because invoker been destroyed after refreshInvoker. This issue may happen in low QPS, 5W QPS happens in high rate(dubbo's default.retires=2 covers the issue).
Note: We have set default.retries=0 in the concerns of idempotency, because consumer will retry always.

The AbstractInvoker throws the RpcException as it been destroyed.

public Result invoke(Invocation inv) throws RpcException {
        if (destroyed.get()) {
            throw new RpcException("Rpc invoker for service " + this + " on consumer " + NetUtils.getLocalHost()
                    + " use dubbo version " + Version.getVersion()
                    + " is DESTROYED, can not be invoked any more!");
        }
}

RegistryDirectory refreshInvoker destroyed the unused invoker, which caused the issue.

private void refreshInvoker(List<URL> invokerUrls) {
        ....
        this.methodInvokerMap = multiGroup ? toMergeMethodInvokerMap(newMethodInvokerMap) : newMethodInvokerMap;
        this.urlInvokerMap = newUrlInvokerMap;
        try {
            destroyUnusedInvokers(oldUrlInvokerMap, newUrlInvokerMap); // Close the unused Invoker
        } catch (Exception e) {
            logger.warn("destroyUnusedInvokers error. ", e);
        }
    }
}

If there is an exception, please attach the exception trace:

Just put your stack trace here!

com.alibaba.dubbo.rpc.RpcException: Rpc invoker for service interface com.ctrip.framework.benchmark.api.BenchmarkService -> dubbo://10.5.89.100:20880/com.ctrip.framework.benchmark.api.BenchmarkService?anyhost=true&application=100013636&async=true&check=false&default.retries=0&default.timeout=1005&dubbo=2.0.1&generic=false&init=true&interface=com.ctrip.framework.benchmark.api.BenchmarkService&methods=echo&pid=45&qos.enable=false&register.ip=10.5.88.92&remote.timestamp=1529581187615&revision=1.0.0&serviceId=framework.soa.dubbobenchmarkservice.v1.dubbobenchmarkservice&side=consumer&timeout=5000&timestamp=1529581726875 on consumer 10.5.88.92 use dubbo version 2.0.1 is DESTROYED, can not be invoked any more!
	at com.alibaba.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:122)
	at com.alibaba.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:73)
	at com.ctrip.framework.cdubbo.internal.filter.CatConsumerFilter.invokeAsync(CatConsumerFilter.java:60)
	at com.ctrip.framework.cdubbo.internal.filter.CatConsumerFilter.invoke(CatConsumerFilter.java:50)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:68)
	at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:74)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:68)
	at com.alibaba.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:53)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:68)
	at com.alibaba.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:47)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:68)
	at com.ctrip.framework.cdubbo.internal.delegate.client.CDubboInvoker.invokeAsync(CDubboInvoker.java:111)
	at com.ctrip.framework.cdubbo.internal.delegate.client.CDubboInvoker.invoke(CDubboInvoker.java:50)
	at com.ctrip.framework.cdubbo.internal.filter.CDubboInvokerFilter.invoke(CDubboInvokerFilter.java:31)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:68)
	at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:52)
	at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:77)
	at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:232)
	at com.ctrip.framework.cdubbo.internal.delegate.client.CDubboClusterInvokerDelegate.invoke(CDubboClusterInvokerDelegate.java:40)
	at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:70)
	at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:51)
	at com.alibaba.dubbo.common.bytecode.proxy0.echo(proxy0.java)
	at com.ctrip.framework.benchmark.client.service.DubboBenchmarkServiceTester.test(DubboBenchmarkServiceTester.java:88)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

[haiyang1985]

@ralf0131 Can you look up the issue?

[haiyang1985]

@chickenlj it happens in 1600 QPS again these days.

[cvictory]

@haiyang1985 please upload a demo that can reproduce this issue?

[cvictory]

Can you upload a demo to reproduce this issue? Haiyang <notifications@github.com> 于2018年7月30日周一 上午10:47写道：

…

@chickenlj <https://github.com/chickenlj> it happens again, in 1600 QPS these days. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2016 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD5mbEVyq3L43YFLDwIRvziedCRAKcNhks5uLnPagaJpZM4VANbg> .

-- Best Regard! cvitory