Application gets stuck when Feign client is used with virtual threads #1364
Description
Steps to reproduce:
- Checkout https://github.com/stsypanov/concurrency-demo
- Run
DependencyApplicationand thenConcurrencyDemoApplication - When both apps are up run
StuckApplicationTest - It will take about 1-2 minutes for the test to complete
- Now go to
demo-service/application.ymland setspring.threads.virtual.enabled: true(by default it'sfalse). - Restart
ConcurrencyDemoApplication - Run
StuckApplicationTestagain
If you now attach a profile (e.g. YourKit) you'll see there's potential deadlock with this stacktrace
Potential deadlock: frozen threads found
It seems that the following threads have not changed their stack for more than 10 seconds.
These threads are possibly (but not necessarily!) in a deadlock or hung.
+-----------------------------------------------------------------------------------------------------------------------------+
| Name |
+-----------------------------------------------------------------------------------------------------------------------------+
| +---Read-Updater Frozen for at least 10s <Ignore a false positive> |
| | | |
| | +---jdk.internal.misc.Unsafe.park(boolean, long) Unsafe.java (native) |
| | | |
| | +---java.util.concurrent.locks.LockSupport.park() LockSupport.java:371 |
| | | |
| | +---java.util.concurrent.LinkedTransferQueue$DualNode.await(Object, long, Object, boolean) LinkedTransferQueue.java:458 |
| | | |
| | +---java.util.concurrent.LinkedTransferQueue.xfer(Object, long) LinkedTransferQueue.java:613 |
| | | |
| | +---java.util.concurrent.LinkedTransferQueue.take() LinkedTransferQueue.java:1257 |
| | | |
| | +---sun.nio.ch.Poller.updateLoop() Poller.java:286 |
| | | |
| | +---sun.nio.ch.Poller$$Lambda.0x0000024081474670.run() |
| | | |
| | +---java.lang.Thread.runWith(Object, Runnable) Thread.java:1596 |
| | | |
| | +---java.lang.Thread.run() Thread.java:1583 |
| | | |
| | +---jdk.internal.misc.InnocuousThread.run() InnocuousThread.java:186 |
| | |
| +---spring.cloud.inetutils Frozen for at least 10s <Ignore a false positive> |
| | | |
| | +---java.net.Inet6AddressImpl.getHostByAddr(byte[]) Inet6AddressImpl.java (native) |
| | | |
| | +---java.net.InetAddress$PlatformResolver.lookupByAddress(byte[]) InetAddress.java:1225 |
| | | |
| | +---java.net.InetAddress.getHostFromNameService(InetAddress, boolean) InetAddress.java:840 |
| | | |
| | +---java.net.InetAddress.getHostName(boolean) InetAddress.java:782 |
| | | |
| | +---java.net.InetAddress.getHostName() InetAddress.java:754 |
| | | |
| | +---org.springframework.cloud.commons.util.InetUtils$$Lambda.0x0000024081187240.call() |
| | | |
| | +---java.util.concurrent.FutureTask.run() FutureTask.java:317 |
| | | |
| | +---java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1144 |
| | | |
| | +---java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:642 |
| | | |
| | +---java.lang.Thread.runWith(Object, Runnable) Thread.java:1596 |
| | | |
| | +---java.lang.Thread.run() Thread.java:1583 |
| | |
| +---Write-Updater Frozen for at least 10s <Ignore a false positive> |
| | |
| +---jdk.internal.misc.Unsafe.park(boolean, long) Unsafe.java (native) |
| | |
| +---java.util.concurrent.locks.LockSupport.park() LockSupport.java:371 |
| | |
| +---java.util.concurrent.LinkedTransferQueue$DualNode.await(Object, long, Object, boolean) LinkedTransferQueue.java:458 |
| | |
| +---java.util.concurrent.LinkedTransferQueue.xfer(Object, long) LinkedTransferQueue.java:613 |
| | |
| +---java.util.concurrent.LinkedTransferQueue.take() LinkedTransferQueue.java:1257 |
| | |
| +---sun.nio.ch.Poller.updateLoop() Poller.java:286 |
| | |
| +---sun.nio.ch.Poller$$Lambda.0x0000024081474670.run() |
| | |
| +---java.lang.Thread.runWith(Object, Runnable) Thread.java:1596 |
| | |
| +---java.lang.Thread.run() Thread.java:1583 |
| | |
| +---jdk.internal.misc.InnocuousThread.run() InnocuousThread.java:186 |
+-----------------------------------------------------------------------------------------------------------------------------+
This isn't a deadlock, but the application gets stuck at this method
public HostInfo convertAddress(final InetAddress address) {
HostInfo hostInfo = new HostInfo();
Future<String> result = this.executorService.submit(address::getHostName);
String hostname;
try {
hostname = result.get(this.properties.getTimeoutSeconds(), TimeUnit.SECONDS);
}
catch (Exception e) {
this.log.info("Cannot determine local hostname");
hostname = "localhost";
}
hostInfo.setHostname(hostname);
hostInfo.setIpAddress(address.getHostAddress());
return hostInfo;
}Apparently, the reason is that executorService is single-thread executor and cannot serve more than 1 request simultaneously:
public InetUtils(final InetUtilsProperties properties) {
this.properties = properties;
this.executorService = Executors.newSingleThreadExecutor(r -> {
Thread thread = new Thread(r);
thread.setName(InetUtilsProperties.PREFIX);
thread.setDaemon(true);
return thread;
});
}I suggest to change switch it to Executors.newCachedThreadPool for it's JavaDocs explicitly says
Creates a thread pool that creates new threads as needed, but will reuse previously constructed threads when they are available. These pools will typically improve the performance of programs that execute many short-lived asynchronous tasks. Calls to execute will reuse previously constructed threads if available. If no existing thread is available, a new thread will be created and added to the pool. Threads that have not been used for sixty seconds are terminated and removed from the cache. Thus, a pool that remains idle for long enough will not consume any resources.