Exponential backoff on failed Elasticsearch node configuration for X-pack monitoring

[ppf2]

Currently, if multiple hosts (https://www.elastic.co/guide/en/beats/filebeat/6.3/configuration-monitor.html#_literal_hosts_literal_3) are specified for X-pack monitoring, if 1 host fails, it will continue to round robin requests to the other host(s). Say we have 2 hosts in the array, if 1 host goes down, it will send all the monitoring requests to the 2nd host until the 1st host returns. During this time, we continue to perform healthchecks against the problem host:

2018-08-14T14:27:44.022-0700	ERROR	pipeline/output.go:74	Failed to connect: X-Pack capabilities query failed with: Get http://localhost:9201/_xpack?filter_path=features.monitoring.enabled: dial tcp [::1]:9201: getsockopt: connection refused

This occurs every 10s until the host returns. For a network with many beat clients, a failure against 1 host in the monitoring hosts array, can result in a lot of additional packets, proportional to the # of beats running in the network. We may want to consider implementing an exponential backoff of these healthchecks against a failed host given this.

[urso]

Checking the code, we indeed there no backoff. Did x-pack monitoring work before? The init loop is supposed to check x-pack availablity every 10s. If init loop succeeds, the beat starts sending.
Assuming a number of beats are started at about the same time, we should also introduce some random jitter, so not all beats will try at about the same time, no matter the backoff strategy.

[ppf2]

Thanks. It worked before, only 1 node became unresponsive on the ES side so it started doing healthchecks against the failed node. Yah, certainly, all the Beats will not be firing off the healthcheck at the exact moment, but if there are enough Beats running on the network, it can still translate to noticeably more requests per second by the network admins.

[ycombinator]

@urso Digging around the beats codebase a bit, sounds like the client implementation in beats monitoring could possible (re)use something like this?

beats/libbeat/outputs/backoff.go

Line 37 in 0053aaf

func WithBackoff(client NetworkClient, init, max time.Duration) NetworkClient {

Just wanted to get your thoughts before I go off and put up a PR. Thanks.

[urso]

The monitoring reporter has 2 phases. First phase: check x-pack monitoring is available, every 10s. If First phase succeeds, the collector phase is started. The WithBackoff can be used for the collector phase.

Not sure about the init phase, though. The WithBackoff wrapper only act on errors. Question: we want to backoff in case x-pack monitoring is disabled in ES (does not generate an error)?

Anyways, WithBackoff reuses backoff from libbeat, as last resort you can use that one.

[ppf2]

we want to backoff in case x-pack monitoring is disabled in ES (does not generate an error)?

If monitoring is disabled in the ES node specified in the hosts array, seems like it will make sense to skip the backoff routine for that host (until it shows up again with the monitoring feature enabled)?