Wavefront output plugin failing with certificate error.

[rushikeshbgithub]

Hi,

I am using wavefront output plugin in telegraf agent to post metrics to wavefront proxy.

I am using nginx proxy in front of both telegraf agent and wavefront proxy.

1. Telegrafs wavefront output plugin performs http://localhost:8071/wavefront/report, nginx receives this request and perform https://... proxy_pass to another nginx sitting before wavefront proxy.
2. After tls, nginx before wavefront proxy again does proxy_pass with http://localhost:2878/report to wavefront proxy listening on localhost 2878.

This used to work until telegraf version 1.23.4 and started failing with telegraf version 1.24.0. Below is error I get in telegraf

error reporting wavefront format data to Wavefront: "Post "https://localhost/wavefront:8071/report?f=wavefront\": x509: certificate is not valid for any names, but wanted to match localhost"

Actually telegraf wavefront plugin should not get https error because ssl part is done only between both nginx, other communication is on plain http.

Everything works fine if telegraf 1.23.4 is used and fails when 1.24.* version is used.
Can someone please help me with this ? Please let me know if you need more information.

[MyaLongmire]

This is the only commit I could find between 1.23.4 and 1.24.*.

@LukeWinikates sorry to ping you but as the author of that pr I am hoping you have some insight :)

[LukeWinikates]

Hi @rushikeshbgithub - quick question. Are you using host and port or are you using url to configure your wavefront output? I think you must have been using host and port, which means that prior to 1.24.x your metrics would have been sent to the wavefront proxy via tcp, but as of the wavefront output changes in the 1.24.x telegraf release, all wavefront metrics are sent via http or https.

I wanted to give you a quick response that might help you with debugging, but I also haven't quite digested your proxy configuration yet. I wonder if maybe using tcp previously, your proxies simply weren't actually involved in the sending of metrics to wavefront. Or is it possible that you have an https url in your wavefront output configuration?

[rushikeshbgithub]

Hi @LukeWinikates, Thanks for your response.
I am using url to configure wavefront, there is no host and port fields added to config.
Below I have put configurations for telegraf wavefront output plugin, nginx configurations.
The reason for using nginx is to offload SSL responsibilty to nginx and let telegraf and wavefront proxy work with http only.
So, tcp is not directly used in communication. We use http interface.
Please let me know if you need any more details from me.

==============================
Telegraf wavefront output plugin configuration

[[outputs.wavefront]]
prefix = "telegraf."
url = "http://localhost:8071/wavefront"
metric_separator = "_"
source_override = ["hostname", "snmp_host", "node_host"]
convert_paths = false
use_regex = false
simple_fields = false

===============================
nginx configuration

1. For nginx on telegraf machine

server {
listen 8071;
server_tokens off;
....
....
location /wavefront {
proxy_ssl_trusted_certificate /etc/certificates/ca-certificates.crt;
proxy_pass https://ip-for-nginx-before-wavefront-proxy:8071/wavefront;
}
....
....
}

2. For nginx on wavefront proxy machine

server {
listen 8071;
server_tokens off;
....
....
location /wavefront {
rewrite ^/wavefront/(.*)$ /$1 break;
proxy_set_header Connection "";
proxy_pass http://localhost:2878;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_buffering on;
}
....
....
}

[Hipska]

@LukeWinikates It seems you fixed the bug upstream, can you also create a PR here to update wavefront module to newest version?

[powersj]

PR to update the dep: #12309 - it would be good to have someone confirm if this fixes this issue.

[powersj]

I merged the dependency update as we are prepping for v1.25.0 for next week. Thanks @LukeWinikates for driving the upstream. If this turns out to not fix the issue, please open a new issue. Thanks!

[pavanrangain]

We were looking to update to 1.24.x for using json_transformation but hitting this bug. Since its a bug introduced only with 1.24.x (of course upstream) can we please look at back porting fix for 1.24.x