Bundle Metricbeat with Elasticsearch

[jasontedor]

For the 8.0.0 release of Elasticsearch, we are aiming to remove monitoring internal collectors/exporters (#47214, #47215). While Metricbeat can now monitor Elasticsearch, the removal of internal collection leaves our users in a position where they have to take extra steps to do such monitoring via Metricbeat. We want to keep the experience of monitoring Elasticsearch simple. To that end, we are aiming to bundle Metricbeat with Elasticsearch and have its lifecycle managed by Elasticsearch. This saves the user from having to install Metricbeat, and also manage its lifecycle.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Packaging)

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Monitoring)

[cachedout]

Hi @jasontedor. Thanks for filing this.

Given that Stack Monitoring can also consume ES logs, do you think it would be worth discussing whether or not to also bundle Filebeat in addition to Metricbeat? It is not required for the monitoring application to function, but provides additional functionality on top of the default metrics. Thoughts?

[beiske]

we are aiming to bundle Metricbeat with Elasticsearch and have its lifecycle managed by Elasticsearch

@jasontedor Does this mean enabling and disabling the bundled Metricbeat be a dynamic setting or will it potentially require a reboot of Elasticsearch?

[jasontedor]

It will be a restart of Elasticsearch. Elasticsearch can't fork a native process after startup is complete, by the security manager, and by our seccomp policy and we are never going to relax that.

[beiske]

Thanks, is the corollary then that Elasticsearch will exit if Metricbeat exits so that both processes may be rebooted?