Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Not sending metrics after stop and start of the StatsdMeterRegistry #1676

Closed
uchandroth opened this issue Oct 29, 2019 · 9 comments · Fixed by #1782
Closed

Not sending metrics after stop and start of the StatsdMeterRegistry #1676

uchandroth opened this issue Oct 29, 2019 · 9 comments · Fixed by #1782
Labels
bug A general bug registry: statsd A StatsD Registry related issue
Milestone
1.4.0

Comments

@uchandroth
Copy link

Environment : spring-boot 2.2.0.RELEASE
micrometer-registry-statsd : 1.3.0 also tried with older versions
steps to replicate the issue:

  1. Once the application is started statsd metrics will be start pumping to stats daemon running on 8125 udp port
  2. invoke StatsdMeterRegistry.stop() --> which will shutdown the udp client
  3. invoke StatsdMeterRegistry.start() --> which will start the udp client. debug mode we can see that udp client status is started
    But statsd metrics will not be send.
    could you please help to understand the issue. Or is this the right way to reinitialise the connection

Background of the issue : we are using a stasd daemon with a dns name, if the deamon is restarted the ip address maybe different, because of the dns caching udp client will not be able to connect to the stasd daemon if the ip address is different.
We have tried setting networkaddress.cache.ttl=10 but it is not getting affected for the StatsdMeterRegistry UdpClient object
@shakuzen shakuzen added the registry: statsd A StatsD Registry related issue label Oct 30, 2019
@shakuzen
Copy link
Member

This looks like a duplicate of #1252

@shakuzen shakuzen added the duplicate A duplicate of another issue label Oct 30, 2019
@shakuzen shakuzen mentioned this issue Oct 31, 2019
Open
@shakuzen shakuzen added bug A general bug and removed duplicate A duplicate of another issue labels Oct 31, 2019
@shakuzen
Copy link
Member

I'd like to verify but I think this will be an issue when stopping and starting the registry even if not changing the address. See #1252 (comment). This will probably be obviated when we stop having the issue with only a single subscriber allowed to the publisher (see #1251 (comment))

@shakuzen shakuzen reopened this Oct 31, 2019
@shakuzen shakuzen changed the title micrometer not sending metrics after stop and start of the udpclient Not sending metrics after stop and start of the StatsdMeterRegistry Oct 31, 2019
@shakuzen shakuzen added this to the 1.4.0 milestone Jan 20, 2020
@shakuzen shakuzen mentioned this issue Jan 20, 2020
Merged
shakuzen added a commit that referenced this issue Jan 27, 2020
@shakuzen @izeye
Rewrite Statsd publishing implementation (#1782)
ae531f2 
Upgrades to the latest version of reactor-core and reactor-netty. Uses a `DirectProcessor` instead of a `UnicastProcessor` that can only be subscribed to once. We need to be able to subscribe multiple times for reconnecting to a server with a new `TcpClient`. For multi-threaded publishing to the `Processor`, we use a `FluxSink` rather than calling `onNext` directly. Starting and stopping the registry should now work as expected and is tested.

Unlike before, there is no buffering done in the `Processor`. If the UDP/TCP client or `lineSink` cannot keep up with the rate metrics are produced at, metric lines will be dropped rather than buffered indefinitely (potentially until the application runs out of memory). A mitigation when using the UDP/TCP client is to use the `BufferingFlux` which is enabled by default - it buffers metric lines in memory up to a configurable size/time before emitting to the UDP/TCP client. As a consequence, the queue size/capacity methods and metrics (`StatsdMetrics`) are no longer available.

Resolves #1212
Resolves #1591
Resolves #1676
Resolves #1741
Supersedes #1251

Co-authored-by: Johnny Lim <izeye@naver.com>
@shakuzen
Copy link
Member

Stopping and starting the StatsdMeterRegistry should work correctly now. Could you try with the latest 1.4.0-SNAPSHOT version and let us know if it is working for you?

@uchandroth
Copy link
Author

micrometer-registry-statsd 1.4.0-snapshot with spring boot 2.2.0.RELEASE, 2.2.4.RELEASE release will throw java.lang.NoClassDefFoundError: io/micrometer/shaded/reactor/core/publisher/FluxSink
FluxSink class was part of the micrometer-registry-statsd 1.3.0 jar.
Could please let me know how can we can integrate micrometer-registry-statsd.1.4.0-snapshot with springboot

@shakuzen shakuzen mentioned this issue Feb 6, 2020
Closed
@shakuzen
Copy link
Member

shakuzen commented Feb 6, 2020

Thanks for trying things out. Indeed it looks like the shaded dependencies are missing from the latest snapshots. I've opened #1843 to fix that. Once that is fixed, if you could try again, things should work.

@shakuzen
Copy link
Member

shakuzen commented Feb 7, 2020

@uchandroth could you try again with the latest snapshots now that a fixed has been merged for the NoClassDefFoundError?

@uchandroth
Copy link
Author

@shakuzen could you please let me know the release version of micrometer-registry-statsd which has this fix

@shakuzen
Copy link
Member

shakuzen commented Mar 5, 2020

For now it is only available in the snapshot version 1.4.0-SNAPSHOT. It will be available in 1.4.0 when that is released.

@uchandroth
Copy link
Author

@shakuzen thanks for the update

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug A general bug registry: statsd A StatsD Registry related issue
Projects
None yet
Milestone
1.4.0
Development

Successfully merging a pull request may close this issue.

Rewrite Statsd publishing implementation micrometer-metrics/micrometer
2 participants
@shakuzen @uchandroth