Deadlock while creating Spring beans with parallel bootstrap threads on IBM Liberty

[calin-jebelean]

Hi,

We have recently migrated to Spring Boot 3.4.x, which creates and initializes beans in parallel upon startup. We are experiencing problems where some of our Spring Boot applications do not even start. We suspect they are stuck in a deadlock related to this parallel creation and initialization of beans.

We see a lot of the following messages in the logs:
Creating singleton bean 'X' in thread "Y" while other thread holds singleton lock for other beans: [Z, ...]

These are the last log messages we see before the application stops logging and eventually runs in the following:
The XYZ application did not issue the ApplicationReadyEvent event in 5 minutes.

Here is an actual example:

[2025-04-06T23:23:45.453+0200] org.springframework.beans.factory.support.DefaultListableBeanFactory : Creating singleton bean 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration' in thread "Default Executor-thread-4" while other thread holds singleton lock for other beans [org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration, springSecurityFilterChain]
[2025-04-06T23:28:13.754+0200] 0000004a SpringBootApp A CWWKC0264W: The application did not issue the ApplicationReadyEvent event in 5 minutes.

We are concerned by the fact that the last bean that is created before the (supposed) deadlock is (according to the log message) org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration, and this bean is also in the list of beans on which a different thread already holds a lock. This does not seem right and this is not a custom bean. We can see a resemblance with https://github.com/spring-projects/spring-framework/issues/34672, so we plan on giving 6.2.6 a try, unfortunately that's not easy for us to do since the issue only manifests itself in production. We are also working on a way to reproduce it in the lower environments. My question would be, is the concern about the WebSecurityConfiguration bean justified or not? Should the fix in 6.2.6 also work in our case or is this potentially a new issue?

Regards
Calin

[jhoeller]

There is indeed a potential for deadlocks in case of fully multi-threaded bootstrap. We had several revisions in that area recently, including #34672. Note that 6.2.6 also has a property spring.locking.strict=true that unconditionally restores pre-6.2 behavior (see #34303 (comment) for the motivation). Both of those are worth testing in your scenario, with the property effectively providing an opt-out that should facilitate your upgrade.

That said, I'd like to understand your scenario better. Which other threads are involved in your bootstrap? Where and when are they set up? (#34303 was about a Pekko actor system, for example.) In a regular bootstrap scenario, there is only one main bootstrap thread, with just an occasional thread potentially started by a specific bean for internal purposes. Generally speaking, broader thread pools should only really become active once the application is initialized - at the very end of the bootstrap phase, with all common beans having been created beforehand.

[jhoeller]

As for the log message about WebSecurityConfiguration, that's not concerning on its own since the additional thread will subsequently await the creation of the WebSecurityConfiguration in the regular singleton lock. However, point taken that the log message is a little misleading. I'll change the wording to "Obtaining singleton bean ..." for 6.2.6.

[calin-jebelean]

There is indeed a potential for deadlocks in case of fully multi-threaded bootstrap. We had several revisions in that area recently, including #34672. Note that 6.2.6 also has a property spring.locking.strict=true that unconditionally restores pre-6.2 behavior (see #34303 (comment) for the motivation). Both of those are worth testing in your scenario, with the property effectively providing an opt-out that should facilitate your upgrade.

That said, I'd like to understand your scenario better. Which other threads are involved in your bootstrap? Where and when are they set up? (#34303 was about a Pekko actor system, for example.) In a regular bootstrap scenario, there is only one main bootstrap thread, with just an occasional thread potentially started by a specific bean for internal purposes. Generally speaking, broader thread pools should only really become active once the application is initialized - at the very end of the bootstrap phase, with all common beans having been created beforehand.

Thank you for the quick feedback. As far as I can tell, we don't have a situation such as the one in #34672, where we're spawning custom threads to deal with bootstraping beans, unless some library is doing that for us. We have normal Spring Boot applications which worked just fine until we started migrating them to Spring Boot 3.4.x and Spring 6.2.x.

The threads that I see in the logs are the following:
background-preinit
Default Executor-thread-3
Default Executor-thread-2
Default Executor-thread-4
Default Executor-thread-5
Default Executor-thread-1
Default Executor-thread-6

These all look like Spring-managed threads to me, reinforcing my belief that we're not spawning custom threads. We'll definitely consider using the new property spring.locking.strict=true, thank you for this suggestion.

Also, can you please give me a hint, why would the deadlock problem not manifest itself when running the application locally? In fact, all log messages produced in the local setup are logged by the "main" thread. The deadlock problem is only manifesting on the actual server, and almost exclusively in the production environment. Is it somehow dependent on the number of processors or processor cores? What could be missing in our local setup in order to see the problem manifesting there as well?

Thank you in advance for taking the time to help us!

Best
Calin

[jhoeller]

The background-preinit thread is Spring Boot managed there but the others are outside of the bootstrap arrangement, maybe coming in from some HTTP endpoint or the like? Are you running on IBM Liberty, by any chance - that's where I see those thread names after some quick googling? The CWWKC error codes also seem to be IBM specific.

Spring itself only uses a single bootstrap thread (typically the JVM launch thread or the Servlet container init thread) and does not actively use others unless explicitly told so. When encountering unexpected calls from other threads during the bootstrap phase, it tries to leniently handle them, but this is not a scenario that we optimize for by default.

In any case, it looks like your Spring application context is being hit by by actual requests before it is fully initialized. The context would automatically use strict locking in a post-bootstrap scenario where such requests are expected (e.g. for lazy-init beans) - whereas during the bootstrap bean initialization phase, it applies lenient locking (as of 6.2).

The spring.locking.strict=true property enforces our stricter post-bootstrap locking model during bootstrap as well which is why it can be used to address such scenarios; this should bypass the problem completely in your case. However, there is still some unusual early hitting of the application context going on there that's worth finding out about.

[calin-jebelean]

Hi Jürgen,

Yes, our applications run on IBM Liberty and you're right, I googled as well and the "Default Executor-thread-X" thread names are indeed managed by Liberty. I didn't know about that and simply meant that our applications themselves do not start such threads explicitly. We'll wait for 6.2.6 and the new property spring.locking.strict=true.

Best
Calin

[calin-jebelean]

I can confirm that the "Default Executor-thread-X" threads in IBM Liberty are involved in creating the beans upon startup. They are also involved in servicing requests later.

Thread-3:
DEBUG --- --- [thread= Default Executor-thread-3] org.springframework.beans.factory.support.DefaultListableBeanFactory : Creating shared instance of singleton bean 'messageDispatcherServlet'

Thread-4:
DEBUG --- --- [thread= Default Executor-thread-4] org.springframework.beans.factory.support.DefaultListableBeanFactory : Creating shared instance of singleton bean 'resourceUrlEncodingFilter'

Thread-6:
DEBUG --- --- [thread= Default Executor-thread-6] org.springframework.beans.factory.support.DefaultListableBeanFactory : Creating shared instance of singleton bean 'soapActionAnnotationMethodEndpointMapping'

I guess these "Creating shared instance" debug logs are unproblematic and reflect the happy case, but as said, we do see deadlocks in production almost with each deployment. We don't have the "liberty" (pun intended) to mess with Liberty's thread pool, as it's managed on the server. Do you plan on doing tests with an IBM Liberty setup, just to see how things hold up in such an environment?

Best
Calin

[jhoeller]

Thanks for the insight, that's very helpful indeed!

It looks like Liberty is bootstrapping some of the Servlet infrastructure (the Spring WS Servlet, the ResourceUrlEncodingFilter) in parallel to the main application context while that context is still being initialized, with each of those Servlet instances and Filter instances initialized in a separate thread but then calling into the main application context for bean retrieval. With strict locking, they would all be blocked until the main application context releases the singleton lock, effectively serializing all those initialization attempts from Servlet/Filter threads. Whereas with lenient locking, they all try to obtain beans within the main context in parallel - which is generally preferable but can lead to different initialization order.

spring.locking.strict=true is definitely going to help for such scenarios, restoring previous locking behavior with all those threads effectively forced to execute serially.

That said, it's very unorthodox for a Servlet container to bootstrap a Servlet application in a multi-threaded way. Traditional Servlet bootstrapping happens in a single thread to begin with. I'll try to find out why Liberty does it that way, even by default apparently, in contrast to Tomcat, Jetty and co.

Any further hints from your side in terms of how the application is concretely deployed on Liberty? A Boot-generated war file or a fat jar with Liberty's custom Spring Boot support (https://openliberty.io/docs/latest/deploy-spring-boot.html), I suppose it's the latter since we get that ApplicationReadyEvent log message from Liberty? Any Liberty-specific deployment hints?

[calin-jebelean]

Any further hints from your side in terms of how the application is concretely deployed on Liberty? A Boot-generated war file or a fat jar with Liberty's custom Spring Boot support (https://openliberty.io/docs/latest/deploy-spring-boot.html), I suppose it's the latter since we get that ApplicationReadyEvent log message from Liberty? Any Liberty-specific deployment hints?

You're right, it's the latter. The last question is difficult for me to answer since we don't do the deployment ourselves but rely on a managed team to handle all the low-level details. Thank you for your support, Jürgen, much appreciated!

[jhoeller]

Alright, I'll see what we can do there. Maybe we can automatically switch to strict locking if we detect an externally driven multi-threaded startup, not requiring an explicit spring.locking.strict=true declaration. It helps to specify the latter in any case though.

The underlying problem is that we only really want to leniently handle threads that are bootstrapped from within the application, for example some custom threads started in specific bean init methods. We never meant to apply lenient locking to multiple external container threads calling into the application bootstrap at the same time. However, those scenarios are hard to differentiate at runtime: the only threads that we can uniquely identify are the main bootstrap thread and Spring-started background init threads; all other threads look the same to us. We need to differentiate between unmanaged (bean-started) and externally managed (container).

[jhoeller]

There is actually one thing that would help still: What's the main bootstrap thread's name in your scenario, Default Executor-thread-1 possibly? Aside from the number suffix, does the main bootstrap thread (that attempts to initialize all singleton beans within the singleton lock) look the same as the additional bootstrap threads that operate outside of the lock, in terms of the thread name?

[calin-jebelean]

There is actually one thing that would help still: What's the main bootstrap thread's name in your scenario, Default Executor-thread-1 possibly? Aside from the number suffix, does the main bootstrap thread (that attempts to initialize all singleton beans within the singleton lock) look the same as the additional bootstrap threads that operate outside of the lock, in terms of the thread name?

The main thread doing the bootstrapping is a random "Default Executor-thread-X". Basically, it appears to randomly pick one thread from the thread pool. I logged it in a @PostConstruct method in one of the application singleton components. Also, for the first time, I managed to reproduce the problem in the local Liberty environment (it won't reproduce in the local Tomcat environment, which is now obvious, because Tomcat has single-threaded bootstrapping).

Log:

... Other bootstrapping logs from DefaultExecutor-thread-6 (the bootstrapping thread in this execution) ...
[      DefaultExecutor-thread-6] [<package-name>.MainThreadComponent: Main thread name: DefaultExecutor-thread-6

[      DefaultExecutor-thread-1] org.springframework.beans.factory.support.DefaultListableBeanFactory: Creating singleton bean 'springSecurityFilterChain' in thread "DefaultExecutor-thread-1" while other thread holds singleton lock for other beans [mvcConversionService, welcomePageHandlerMapping, mvcHandlerMappingIntrospector, org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration, mvcHandlerMappingIntrospectorRequestTransformer]
[      DefaultExecutor-thread-1] org.springframework.beans.factory.support.DefaultListableBeanFactory: Creating singleton bean 'mvcHandlerMappingIntrospector' in thread "DefaultExecutor-thread-1" while other thread holds singleton lock for other beans [mvcConversionService, welcomePageHandlerMapping, mvcHandlerMappingIntrospector, org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration, mvcHandlerMappingIntrospectorRequestTransformer, springSecurityFilterChain]

Thread 6 is doing the bootstrapping (in this execution), including running the @PostConstruct method in MainThreadComponent (third log line), which prints the name of the bootstrapping thread. The last 2 logs were produced by a manually triggered request while the bootstrapping was still running, which blocked the whole application. This was inspired by your suggestion that the application was trying to service requests while being initialized and I could reproduce the problem locally as soon as I bombarded the locally starting application with requests. We might be able to fix the issue by simply not routing requests to an instance which is currently starting and I'm honestly not sure why this isn't the status quo. I would have to contact the specialized team with this suggestion.

[jhoeller]

Thanks, that makes the picture even more complete.

For the record, I consider it wrong in Liberty's Spring Boot deployer to propagate requests to the application while it is not fully bootstrapped and therefore not fully ready yet. That's another difference in Tomcat: not only is there only a single bootstrap thread there, the application is not going to receive requests before the bootstrap ended. It's the same in Jetty and Undertow, and I suspect that when using Liberty's standard web application deployer for .war files it might behave like Tomcat as well.

Tightening this in the request routing setup of your system might be a solution indeed. I just feel that Liberty itself should provide that guarantee, not accepting/propagating requests before the app is fully deployed - just like Tomcat and Jetty.

Nevertheless, I am going to see what we can do to make our internal assumptions more defensive in such scenarios where we are being hit by a container thread pool early, without requiring an explicit spring.locking.strict=true declaration.