StatsD metric publishing stopping unexpectedly with Micrometer 1.3.14

[breun]

Multiple teams in my company have reported that StatsD metrics stop getting published at unexpected moments since they upgraded to Micrometer 1.3.14 (Spring Boot 2.2.10.RELEASE) from 1.3.11 (Spring Boot 2.2.9.RELEASE). Restarting an application brings metrics to life again for a while, until they disappear again.

We’ve also seen multiple applications stopping StatsD metrics publishing at the same time, which suggests this might be triggered by something external to the applications, like a network hiccup.

We’ve also seen issues like this one in Micrometer in the past, e.g. #1591. The shaded Reactor and Reactor Netty dependencies were upgraded in 1.3.14, which were also involved in previous similar issues.

We've let our teams downgrade Micrometer to 1.3.11, and the issue hasn't been observed since.

[shakuzen]

I suspect this has to do with upgrading the shaded versions of reactor-netty #2263. If we cannot figure out a way to avoid the problem mentioned in this issue with the latest patch versions, we will have to roll back the upgrade of reactor-netty, or we will have to backport the changes from 1.5.x, including upgrading the minor version of reactor/reactor-netty (though they're shaded, so effectively no impact to users expected).

[shakuzen]

I have reverted the upgrade of the shaded reactor/reactor-netty/netty dependencies, which is an unfortunate solution, but it presents less risk and change compared to backporting the changes from 1.5.x, which would be required to keep the behavior we want on the latest patch versions in 1.3.x. Unfortunately we were depending on behavior that was a bug, and to properly implement the retry logic requires bigger changes.

@breun 1.3.15 will be released soon but feel free to try the snapshots with the dependency upgrade reverted if you have a chance to confirm the issue is resolved by this.