Metricbeat xpack monitoring cannot deal with trailing slash on the url

[philippkahr]

Beats: 7.14.1
MacOS 11.5.2 and Windows 10.

Steps to reproduce:

1. Start a cluster with docker
2. Configure metricbeat to use the elasticsearch-xpack.yml module
3. For hosts: ["http://localhost:9200"] change to hosts: ["http://localhost:9200/"] so it features a trailing /.
4. Observe the 404 and 400 errors coming in.

2021-09-08T09:19:46.039+0200    INFO    module/wrapper.go:259   Error fetching data for metricset elasticsearch.shard: HTTP error 400 in : 400 Bad Request
2021-09-08T09:19:46.040+0200    INFO    module/wrapper.go:259   Error fetching data for metricset elasticsearch.cluster_stats: HTTP error 404 in : 404 Not Found
2021-09-08T09:19:46.047+0200    INFO    module/wrapper.go:259   Error fetching data for metricset elasticsearch.ml_job: HTTP error 400 in : 400 Bad Request
2021-09-08T09:19:46.047+0200    INFO    module/wrapper.go:259   Error fetching data for metricset elasticsearch.enrich: HTTP error 404 in : 404 Not Found

Steps to fix:

1. Remove the trailing / slash
2. See monitoring data coming in.

Expected behaviour:

• Beats should deal with the trailing / or throw an error on startup that an invalid configuration was detected.

[philippkahr]

If I am not mistaken e.g. the line

beats/metricbeat/module/elasticsearch/elasticsearch.go

Line 163 in 76974f8

content, err := fetchPath(http, uri, "/_nodes/_local/nodes", "")

automatically appends /_nodes to the settings from the hosts e.g. http://localhost:9200//_nodes whilst the

beats/metricbeat/module/elasticsearch/elasticsearch.go

Line 183 in 76974f8

content, err := fetchPath(http, uri, "_cluster/state/master_node", "")

appends _cluster which results in a correct http://localhost:9200/_cluster.

I guess we need to parse the hosts properly and verify if appending a / is necessary.

[elasticmachine]

Pinging @elastic/stack-monitoring (Stack monitoring)

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[jsoriano]

May be related or caused by #27842

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[mhert]

I just ran into this issue this weekend, and it took some time, until I found out that I had to remove the trailing slash.

I'm pretty sure I'm not the first one, who ran into this, so I would appreciate it if this could get some priority.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!