Document using Metricbeat for collecting and shipping monitoring metrics

[lcawl]

Related to elastic/beats#7035

• Update overview (benefits of new method) in https://www.elastic.co/guide/en/elastic-stack-overview/master/xpack-monitoring.html

• Update architectural diagrams (include Metricbeat) in https://www.elastic.co/guide/en/elastic-stack-overview/master/how-monitoring-works.html See https://github.com/elastic/Design/issues/1090 and [DOCS] Updated monitoring architecture diagrams #125

• Add new MetricBeat installation, configuration steps for Kibana data collection in https://www.elastic.co/guide/en/elastic-stack-overview/master/monitoring-production.html

• Add new MetricBeat installation, configuration steps for all other products' data collection in https://www.elastic.co/guide/en/elastic-stack-overview/master/monitoring-production.html, as that functionality is enabled.

• Update the monitoring pages in the Kibana Reference to align with these changes (e.g. https://www.elastic.co/guide/en/kibana/master/monitoring-xpack-kibana.html) See [DOCS] Update Kibana monitoring tasks kibana#23736

• Update the monitoring pages in the Elasticsearch Reference to align with these changes (e.g. https://www.elastic.co/guide/en/elasticsearch/reference/master/configuring-monitoring.html). See [DOCS] Updates Elasticsearch monitoring tasks elasticsearch#34339

• Update the monitoring pages in the other product references to align with these changes (e.g. https://www.elastic.co/guide/en/logstash/current/configuring-logstash.html, https://www.elastic.co/guide/en/beats/filebeat/master/monitoring.html), as that functionality is enabled.

[ycombinator]

I apologize for missing this in the PR review, but looking at the "beta" note for monitoring Kibana above the new diagram in https://www.elastic.co/guide/en/elastic-stack-overview/master/how-monitoring-works.html, it should say 6.4.0 instead of 6.5.0.

I also just realized that we've pushed out the new diagram + "beta" note to the 6.x docs (https://www.elastic.co/guide/en/elastic-stack-overview/6.x/how-monitoring-works.html) but we haven't shown users how to go about installing metricbeat and configuring it+Kibana to achieve the architecture shown in the diagram (item 3 in the checklist above). If you could point me to the right repo/files, I can take a shot at this and put up a PR for your review.

Finally, just a heads up that we are getting close to having Metricbeat monitor Elasticsearch. It should be ready for the 6.5.0 release. You can follow along on our progress in elastic/beats#7035. This will require another update to the architecture diagram and also docs for showing users how to install metricbeat and configure it+Elasticsearch to achieve the new architecture.

[lcawl]

Thanks @ycombinator I've updated the beta note in master and 6.x branches via fa3dd33

Does this mean the new diagram should be retrofitted to the 6.4 branch too?

[ycombinator]

Does this mean the new diagram should be retrofitted to the 6.4 branch too?

Yes, please!

[lcawl]

If more products are updated to use metricbeat for monitoring, the appropriate documentation will be updated in new PRs.