Beats central monitoring Phase 1

[monicasarbu]

When you deploy a large number of Beats, it becomes challenging to monitor the Beats itself.
A solution would be for the Beats to report health status to a collection point, such as Elasticsearch, and visualize it with Kibana.

The following health metrics should be sent to Elasticsearch:

• libbeat.beat.cpu_usage
• libbeat.beat.memory_usage
• libbeat.publisher.published_events
• libbeat.publisher.messages_in_worker_queues
• libbeat.outputs.acked_events Export health metrics via expvar #3423
• libbeat.outputs.messages_dropped
• libbeat.outputs.send_bytes Export health metrics via expvar #3423
• libbeat.outputs.failures Export health metrics via expvar #3423

Each Beat exports more metrics via expvar, but it should send only a subset of these metrics to Elasticsearch.

By default, the health metrics are sent directly to the Elasticsearch cluster configured in the outputs.elasticsearch, but you can also configure an extra Elasticsearch cluster to send the monitoring data to.

TODO:

• Differentiate between the metrics exported via expvar to send only a subset
• Send the health metrics to Elasticsearch

Configuration:

monitoring:
    enabled: true
    period: 10s
    elasticsearch: ["localhost:9201"]

UPDATE: The CPU usage is exported under different fields. See #3422

cc-ed @bohyun-e @brandonmensing

[ruflin]

As reference, here is the old issue where this all started: #463

[bohyun-e]

enabled: true

Would the default value be true here? or false?

period: 10s

I'm not 100% sure what happens when you have the collection period that is different from the ES monitoring collection interval. But reading from the doc, my gut feeling is that whatever ES's collection interval is - should be applied in other products, such as Kibana Monitoring collection interval. I'm guessing it would be the same for Beats, but it'd be a good idea to confirm.

[uboness]

I'm not 100% sure what happens when you have the collection period that is different from the ES monitoring collection interval. But reading from the doc, my gut feeling is that whatever ES's collection interval is - should be applied in other products, such as Kibana Monitoring collection interval. I'm guessing it would be the same for Beats, but it'd be a good idea to confirm.

I don't think we should have this restriction (that all collection intervals are equal). I also don't know how we can even enforce it.

Different systems may need different intervals, and the monitoring UI should deal with that.

[lswith]

It would be extremely nice to simply have one more commandline option to turn on only expvar variables, rather than the httpprof commandline option. This would allow other tools to scrape each beat type on their own interval.

[bohyun-e]

cc: @pickypg @tsullivan @skearns64

[valentin-fischer]

Hi,

Any progress on this ? Is there a way to export this as json and not send everything to elasticsearch ?

Thank you!