With 1.26.2 update wavefront output plugin fails with 404 error if the URL has path

[pavanrangain]

Relevant telegraf.conf

[[outputs.wavefront]]
  url = "http://localhost:29999/wavefront-proxy"
  metric_separator = "."
  source_override = ["hostname", "agent_host", "node_host"]
  convert_paths = true
  immediate_flush = true

Logs from Telegraf

wavefront flushing error: error reporting wavefront format data to Wavefront. status=404

System info

Telegraf 1.26.2, Docker 20.10.14

Docker

No response

Steps to reproduce

1. Have a reverse proxy like nginx/tarefik which forwards a particular URL path to wavefront proxy (strips the base path before forwarding)
2. Have telegraf configured with output config as above (change the host and port part to that of the proxy)
3. Run telegraf
   ...

Expected behavior

telegraf should be able to forward metrics to wavefront proxy through the reverse proxy

Actual behavior

telegraf is failing to send metrics and erroring out with 404

Additional info

This started happening with 1.26.2 update where Wavefront sdk is updated to 0.12.0. With 1.26.1 we see no issues
We have upstream issue created wavefrontHQ/wavefront-sdk-go#133

[powersj]

Thanks for the issue and the upstream issue too. If you do not see any traction on the upstream PR can you ping us here again and we can revert the change to go back to the previous version.

Thanks again!

[LukeWinikates]

at first glance, this looks like it might be the same issue as #11925

I'll take a closer look later today and follow up with either a suggested configuration change or an update about and telegraf sdk fixes that might be needed.

@pavanrangain thanks for reporting this

[LukeWinikates]

@pavanrangain

Thanks for opening this PR and identifying the upstream issue. My goal is to implement a fix in the Wavefont SDK and get a release out this week - by May 5th. @powersj I can follow up here after we get that upstream release out.

[LukeWinikates]

@pavanrangain the PR linked to this thread should fix your issue.

One thing I noticed in your example: your wavefront config says host = "http://localhost:29999/wavefront-proxy", but I think that you must be using the url field. I wouldn't expect host to work at all with a url string.

[pavanrangain]

@LukeWinikates Thanks for quick fix. My bad wrongly gave host instead of url in the example. We are using url itself.
Fixed the example

[pavanrangain]

Hi @powersj Any timeline on the next release with this fix ?

[powersj]

The dependency bump will go into v1.26.3 on Monday.