Description
At work we have noticed that one of our applications have started leaking memory. After diagnosing with VisualVM over a period of time, I have noticed that the heap size of io.prometheus.client.CKMSQuantiles$Item
is gradually increasing over time without it ever lowering.
Using VisualVM, here is a screenshot of the first snaphot that we did
And here is the second, around an hour later
As you can see, the amounts of int
's have also increased which seem to correspond to the members of the Quantiles Item
class.
Although its expected that the number of items in the quantile increase, in our case the memory is never freed and our applications actually restart due to OOM roughly once a day. Our bucket size by default is 10 minutes, so it should be freeing up the memory shortly after that point.
We are btw using https://developer.lightbend.com/docs/cinnamon/current/plugins/prometheus/prometheus.html which uses client_java under the hood, you can also see the default configuration settings.