Remove virtual thread support for Undertow as it leaks memory

[Tythor]

Hello,

I am observing an issue with virtual threads in Undertow in Spring Boot v3.3.0-M2. When using spring.threads.virtual.enabled=true in the application.properties or

@Bean
public UndertowDeploymentInfoCustomizer undertowDeploymentInfoCustomizer() {
 return deploymentInfo -> deploymentInfo.setExecutor(Executors.newVirtualThreadPerTaskExecutor());
}

as a configuration, the memory usage of the application increases dramatically under load. I performed a wrk test with the command wrk -d30 -t10 -c150 http://localhost:8080 and my application's memory usage increased to 20GB in just 30 seconds.

Interestingly, when using a profiler, I observed that the JVM reported significantly lower memory usage than what the activity monitor was showing.

The application eventually crashes with an OOM error:

2024-02-29T19:08:44.404-08:00 ERROR 91152 --- [ndertow-1044448] io.undertow.request                      : UT005023: Exception handling request to /

jakarta.servlet.ServletException: Handler dispatch failed: java.lang.OutOfMemoryError: Cannot reserve 16364 bytes of direct buffer memory (allocated: 17179860008, limit: 17179869184)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1104) ~[spring-webmvc-6.1.4.jar:6.1.4]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979) ~[spring-webmvc-6.1.4.jar:6.1.4]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) ~[spring-webmvc-6.1.4.jar:6.1.4]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903) ~[spring-webmvc-6.1.4.jar:6.1.4]
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:527) ~[jakarta.servlet-api-6.0.0.jar:6.0.0]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) ~[spring-webmvc-6.1.4.jar:6.1.4]
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614) ~[jakarta.servlet-api-6.0.0.jar:6.0.0]
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at com.example.demo.RequestFilter.doFilter(RequestFilter.java:19) ~[main/:na]
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-6.1.4.jar:6.1.4]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.4.jar:6.1.4]
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-6.1.4.jar:6.1.4]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.4.jar:6.1.4]
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-6.1.4.jar:6.1.4]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.4.jar:6.1.4]
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) ~[undertow-core-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) ~[undertow-core-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) ~[undertow-core-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:276) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:132) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:256) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:101) ~[undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393) ~[undertow-core-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:859) ~[undertow-core-2.3.12.Final.jar:2.3.12.Final]
at java.base/java.lang.VirtualThread.run(VirtualThread.java:309) ~[na:na]

Output of java -version

openjdk version "21.0.2" 2024-01-16
OpenJDK Runtime Environment GraalVM CE 21.0.2+13.1 (build 21.0.2+13-jvmci-23.1-b30)
OpenJDK 64-Bit Server VM GraalVM CE 21.0.2+13.1 (build 21.0.2+13-jvmci-23.1-b30, mixed mode, sharing)

I was able to reproduce this behavior when running the application in a Docker container, when compiling to a native-image, on different JDKs (OpenJDK, GraalVM, and Corretto), and on different macOS architectures (Intel and Apple silicon).
Tagging related issue #38819.

Thank you.

[mhalbritter]

That's one of the drawbacks when moving from a thread pool to virtual threads per task without a pool. When using thread pools, they usually have an upper bound for the size, limiting the maximum resource consumption. Unpooled virtual threads don't have that. I assume the application you're benchmarking is a simple demo application which just returns some hardcoded data? Because usually, if there's no pool in the webserver, some other pool in the application is limiting the throughput (e.g. the connection pool to the database).

Btw, you don't see the memory usage in VisualVM, as this is direct allocated memory, which is off-heap.

[Tythor]

Yes, it's the Spring Boot demo project with a basic GET endpoint with no return value. Sorry, could you explain the cause for this again? Aren't virtual threads still limited by the size of the platform threads in the carrier pool? The Tomcat and Jetty servlets don't seem to have the same issue when using virtual threads. Also, thank you for the explanation about the profiler.

[mhalbritter]

One possibility could be this:

1. VT1: Start accepting the request
2. VT1: Allocate memory
3. VT1: Block on something (maybe reading http from the TCP socket)
4. VT1: Handle request
5. VT1: Free memory

With enough requests lined up, the block at 3. could be the reason why the memory is exhausted. At this block, the scheduler frees the underlying platform thread, and switches to a new virtual thread, which allocates memory, blocks, a new virtual thread is scheduled, allocates memory, etc etc.

[mhalbritter]

You could use a SimpleAsyncTaskExecutor with setConcurrencyLimit to limit the maximum allowed number of virtual threads.

[MetaiR]

as long as I remember, there is an issue that talks about virtual threads in Spring Boot here:
#38819

there is one part I saw that says in version 3.3.x the usage of the virtual thread must automatically happen for undertow. so can it be related to this issue?

can u check if this also happens with version 3.2.x ? @Tythor

[Tythor]

@MetaiR
Yes, I can confirm the same behavior happens in v3.2.3. Since spring.threads.virtual.enabled=true does not apply to Undertow servlets until v3.3.0, I had to use the undertowDeploymentInfoCustomizer to enable virtual threading.

@mhalbritter
I attempted to use a SimpleAsyncTaskExecutor with virtual threads with a concurrency limit of 10, but the issue still persisted.

@Bean
public UndertowDeploymentInfoCustomizer undertowDeploymentInfoCustomizer() {
    return deploymentInfo -> {
        SimpleAsyncTaskExecutor simpleAsyncTaskExecutor = new SimpleAsyncTaskExecutor();
        simpleAsyncTaskExecutor.setVirtualThreads(true);
        simpleAsyncTaskExecutor.setConcurrencyLimit(10);
        deploymentInfo.setExecutor(simpleAsyncTaskExecutor);
    };
}

However, this led me to try using a non virtual threaded executor instead. Surprisingly, I observed the same behavior, albeit at a much slower growth rate of about 2m for 20GB instead of 30s. These three executors produced similar results:

deploymentInfo.setExecutor(new SimpleAsyncTaskExecutor());
deploymentInfo.setExecutor(Executors.newThreadPerTaskExecutor(Thread.ofPlatform().factory()));
deploymentInfo.setExecutor(Executors.newThreadPerTaskExecutor(Executors.defaultThreadFactory()));

However, when using

deploymentInfo.setExecutor(Executors.newCachedThreadPool());

the memory growth disappeared. Looking into the implementation, the main difference is that the cachedThreadPool executor has a keepAliveTime of 60s, while the others have a keepAliveTime of 0s.

Reducing the keepAliveTime to as low as 1s did not cause the memory growth in my tests. But setting it to 0s reproduced the same issue.

deploymentInfo.setExecutor(new ThreadPoolExecutor(0, Integer.MAX_VALUE, 1L, TimeUnit.SECONDS, new SynchronousQueue<>()));

So it looks like this issue is not unique to virtual threads, but perhaps with the way Undertow is handling executors?

[mhalbritter]

Yeah, looks like something is wrong here, but it doesn't seem to be in Spring Boot. Please open an issue on the Undertow tracker, and feel free to drop the link in this issue. Thanks!