Allow health indicators to be run asynchronously with the health endpoint providing a view of their current state

[rfelgent]

Hello,

it would be great that spring boot actuator provides a locking/semaphore/mutex logic out-of-the box to prevent its endpoints from flooding.
I have the "actuator/health" especially in mind, as this endpoint is extendable by custom HealthIndicators with potential problems. One of my HealthIndicators caused a long running task by invocing another service/repository (in the end: a missing response/connection timeout fixed the trouble to some degree). As the calculation took so long, my monitoring system (consul) fired more requests in order to get a state. But that only flooded my system...

Currently, there is a cache logic implemented for the actuator endpoints: once the value is calculated it will be cached. The missing piece for me is the semaphor/lock mechanism for not yet calculated (or obsolete/invalidated) values of actuator endpoints. This feature would help to avoid a flooding of the system.

I was wondering if such a feature could be provided by SpringBoot Actuator (like the caching...) and wanted to reach you out before implementing a custom solution on my own...

Best regards.

[wilkinsona]

Something like this would help with ActiveMQ's failover support where checking the health of a connection can block indefinitely. See #25188 (comment) for some details.

[rfelgent]

Hi @wilkinsona,

I will start to implement something like this and will get back to you as soon as the first draft is ready

[wilkinsona]

@rfelgent Just to set expectations, my comment was primarily intended just to link the two issues together. It doesn't necessarily mean that we'll decide to add this sort of functionality to Spring Boot. As I said in my comment on #25188, it'll add quite a bit of complexity and we'll need to weigh up the pros and cons. There are also a few different approaches that could be taken. If you'd like to implement something and share it with us, that would be great. I just want you to be aware that we may decide not to do anything in this area or decide to take a different approach to whatever you may propose.

[rfelgent]

Hi @wilkinsona ,

thx for clearifying "my expectation". :-)

My main goal was to ensure that I did not miss this feature in spring actuator itself.
It seems that this feature is not avialable at the moment and that it might be valuable for others as well (thx for the link to issue #25188)

Anyway, I prefer a lock/semaphore feature provided by Spring Boot Actuator, as it would save me from managing custom code in my app or a custom dependency.

I would like to talk about the functionality in general and my ideas of an implementation with somebody from Spring Boot team, so that I can push that topic.
I am totally fine if either the functionality or its implementation is rejected - but we should talk about it first, shouldn't we ?

Best regards

[wilkinsona]

From #2652:

I've been wondering about something that runs the health indicators periodically in the background and the health endpoint would then just respond immediately with the current aggregated status.

[rfelgent]

Hi @wilkinsona ,

sorry for my abscence last days!

I have noticed, that you have changed the topic: My original idea of a "mutex/lock" functionality is dropped in favour of an asynchronous, periodically running task ?

Can I conclude, that this is the favourite design/concept ?

How would this concept prevent the endpoint (or its calculation) from flooding ?

[wilkinsona]

We were imagining that a request to the health endpoint would immediately return the current state and something else would keep this current state up-to-date. This could be done by scheduling something that runs periodically in the background, or on demand when a request is received. In the on-demand case, we'd probably need a configurable maximum period that a request will wait for the health status to be updated before it responds with down or unknown.

[rfelgent]

does the concept include a dedicated scheduling for each healthcontributor with specific rate and/or delay?

[wilkinsona]

I'm afraid we don't know yet, hence the pending design work label. My initial reaction is that we'd want to avoid that level of complexity and, if we took a scheduling-based approach, we'd want to use one schedule for all of the contributors.

[gmax0]

Is the design work still pending?

[wilkinsona]

Yes, it is. We may take a look at it in the next month or two as part of planning Spring Boot 2.6.

[denisa]

Could https://github.com/antoinemeyer/spring-boot-async-health-indicator serves as inspiration?

[wilkinsona]

Thanks, @denisa. There has been some discussion of that on #2652.