NaN for tomcatServletRequest metrics export to Graphite

[mpaluchowski]

We're seeing errors with Micrometer 1.0.0 exporting Tomcat metrics to Graphite, on Spring Boot 1.5.10. Here's the output from Graphite's Carbon logs:

[2018-02-21T12:27:37.437+0100] INFO [tcp] parse failed {"error": "bad message: \"services.bp.file-storage.dev.all.file-storage-4289024796-e8bq3.tomcatServletRequest.name.default.mean NaN 1519212457\\n\"", "peer": "10.149.18.190:37792"}
[2018-02-21T12:27:37.437+0100] INFO [tcp] parse failed {"error": "bad message: \"services.bp.file-storage.dev.all.file-storage-4289024796-e8bq3.tomcatServletRequest.name.default.p50 NaN 1519212457\\n\"", "peer": "10.149.18.190:37792"}

It's only for the tomcatServletRequest.name.default.p50 and tomcatServletRequest.name.default.mean metrics. Others in the namespace export without errors.

Out configuration is:

management.metrics.export.graphite.protocol=plaintext
management.metrics.export.graphite.step=PT10S

and beyond we have a custom implementation of HierarchicalNameMapper, that adds the prefix of storage.dev.all....

Is that a bug? Any idea what could be causing it?

[jkschneider]

Looks like we need to shunt FunctionTimer mean and percentiles to 0 when there are no samples coming in (divide by 0 results in an NaN here I think). However, Gauges produce NaN when the object they are sampling becomes null as a feature, so perhaps we want to NOT ship any NaN Gauge to Graphite.

Would you mind sharing your carbon configuration?

When I push NaN producing gauges to Graphite, they don't appear in the catalog of metrics (as expected), but there is no error in the carbon logs. Trying to see what you see.

Hi, I'm working with @mpaluchowski and will be probably able to provide info you need. Could you specify, if possible, what information exactly would you like to get from carbon configuration? All of it? :) We are using go-carbon.

[jkschneider]

Not sure what I'm looking for. Never really operationalized Graphite before. This is the conf I'm using. Any obvious differences?

[jkschneider]

Ah, never mind. I bet go-carbon is just more prescriptive since its a port of the original. I'll check that out. Thanks!

[jkschneider]

Still couldn't reproduce with go-carbon, but I believe the underlying issue has been addressed correctly.