[BUG] BlobContainerClient creates threads that aren't stopped on shutdown

[KatieKoslosky]

Describe the bug
Since upgrading from Sprint Boot 2.5.6 to Spring Boot 2.7.9, I see numerous warning on application shutdown notifying me that threads have not been properly stopped. Testing showed this only occurred after the usage of BlobContainerClient, and I was able to produce a minimal example
I wonder if it's similar to #33873?

Exception or Stack Trace
2023-03-24 15:32:07.889 WARN 13988 --- [n(29)-127.0.0.1] o.a.c.loader.WebappClassLoaderBase : The web application [ROOT] appears to have started a thread named [parallel-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
2023-03-24 15:32:07.890 WARN 13988 --- [n(29)-127.0.0.1] o.a.c.loader.WebappClassLoaderBase : The web application [ROOT] appears to have started a thread named [parallel-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
2023-03-24 15:32:07.890 WARN 13988 --- [n(29)-127.0.0.1] o.a.c.loader.WebappClassLoaderBase : The web application [ROOT] appears to have started a thread named [parallel-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)

To Reproduce
Start the server. Run the code. Stop the Server. Messages are on application shutdown

Code Snippet

@configuration
public class AzureStorageBlobContainerClientConfig {

@Value("${blob.connection-string}")
String connectionString;

@Value("${blob.container-name}")
String containerName;

/**
 * Builds the Azure Blob Container client to connect to Azure Blob Storage container 
 * 
 * @return A BlobContainerClientBuilder object 
 */
@Bean
public BlobContainerClient getContainerClient() {
	BlobContainerClientBuilder client = new BlobContainerClientBuilder();
    client.connectionString(connectionString);
    client.containerName(containerName);
    return client.buildClient();
}

}

@RestController
public class GreetingController {

@Autowired
BlobContainerClient containerClient;

@GetMapping("/test")
public String test() {
	ListBlobsOptions options = new ListBlobsOptions()
			.setPrefix("test/")
			.setDetails(new BlobListDetails()
					.setRetrieveDeletedBlobs(false)
					.setRetrieveSnapshots(false)
					.setRetrieveTags(true));
	
	Duration timeout = Duration.ofSeconds(5);
	
	for (BlobItem blob : containerClient.listBlobs(options, timeout)) {
		//iterate through list
	}
	return "Test";
}

}

Setup (please complete the following information):

• Library/Libraries: com.azure:azure-storage-blob:12.21.1
• Java version: 8
• Frameworks: spring-boot-starter-web:2.7.9

[joshfree]

Thanks for filing this github issue, @KatieKoslosky. Someone from the storage team will follow up shortly.

[alzimmermsft]

Thanks for filing this @KatieKoslosky, and this does look to be similar to #33873.

The threads that aren't stopped are created by Reactor's global Schedulers, in this case you're seeing the Schedulers.parallel(). Given these threads are from a shared global context I don't want to have a shutdown hook for them in the SDKs as we can't guarantee this won't affect other code running in an environment, so we'll need to take a deeper look at our usage of these shared Schedulers to find the best solution to this issue, and this may take a while given the complexity. I do have a few questions though to get a better understanding of what's going on:

• When you were using Spring Boot 2.5.6 were you also using the Storage, or other Azure, SDKs? Usage of the shared Schedulers has existed for a while and it's possible these threads worked the same in that environment but there wasn't any logging around it.

• As part of the upgrade to Spring Boot 2.7.9 was the version of the Storage, or other Azure, SDKs upgraded as well?

• Do you think it would be possible to create a simple reproduction using just Reactor where a simple operation uses one of their shared global Schedulers to see if this is broader than just the Azure SDKs.

[KatieKoslosky]

Thank you, @alzimmermsft for your response!

• When you were using Spring Boot 2.5.6 were you also using the Storage, or other Azure, SDKs? Usage of the shared Schedulers has existed for a while and it's possible these threads worked the same in that environment but there wasn't any logging around it.

Yes - we are using the same Storage Account and the same code that we were before the Spring upgrade.
With the limited code example I produced for this ticket, I do see the error when I use Spring Boot 2.7.9, but I do not see the error when I use Spring Boot 2.5.6

• As part of the upgrade to Spring Boot 2.7.9 was the version of the Storage, or other Azure, SDKs upgraded as well?

Initially, no. We were on azure-storage-blob version 12.13.0 before the Spring upgrade.
Upon noticing these shutdown errors, I decided to upgrade to the newest version 12.21.1.
The shutdown errors occur with both these versions of the azure dependency.

• Do you think it would be possible to create a simple reproduction using just Reactor where a simple operation uses one of their shared global Schedulers to see if this is broader than just the Azure SDKs.

I'm not super familiar with Schedulers, but is this the kind of minimal example you were looking for?

	@GetMapping("/greeting")
	public Flux<Integer> greeting(@RequestParam(value = "name", defaultValue = "World") String name) {
		Flux<Integer> flux = Flux.range(0, 2)
                .publishOn(Schedulers.parallel());
		return flux;
	}

If so, I can say I see the same messages on shutdown if this was invoked

2023-03-30 11:35:57.882  WARN 3632 --- [on(7)-127.0.0.1] o.a.c.loader.WebappClassLoaderBase       : The web application [ROOT] appears to have started a thread named [parallel-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 java.lang.Thread.run(Thread.java:750)
2023-03-30 11:35:57.883  WARN 3632 --- [on(7)-127.0.0.1] o.a.c.loader.WebappClassLoaderBase       : The web application