http_response plugin have issues when urls list contains IPv4 and IPv6 addresses

[zak-pawel]

Issue is present only when interface is set to proper value in telegraf.conf

Relevant telegraf.conf for Windows:

[[inputs.http_response]]
  ## List of urls to query.
  urls = ["http://127.0.0.1:15000/hello", "http://[::1]:15000/hello"]

  ## Interface to use when dialing an address
  interface = "Loopback Pseudo-Interface 1"

Relevant telegraf.conf for Linux:

[[inputs.http_response]]
  ## List of urls to query.
  urls = ["http://127.0.0.1:15000/hello", "http://[::1]:15000/hello"]

  ## Interface to use when dialing an address
  interface = "lo"

System info:

1. Telegraf 1.16.2 (but probably there is problem since interface was added to plugin here: Add support for interface field in http_response input plugin #6006 - so from Telegraf 1.12.0)
2. Windows 10 Pro and Ubuntu 18

Steps to reproduce:

1. Start web server which listens on the TCP6 and TCP4 networks. For example:

	http.HandleFunc("/hello", func(writer http.ResponseWriter, request *http.Request) {
		fmt.Fprintf(writer, "hello\n")
	})
	http.ListenAndServe(":15000", nil)

2. Verify that web server responds successfully for http://127.0.0.1:15000/hello and http://[::1]:15000/hello (using web browser or curl)
3. Run Telegraf in Windows with given config (be sure that proper name of loopback interface is used).
4. Run Telegraf in Linux with given config (be sure that proper name of loopback interface is used).

Expected behavior:

1. Web server starts successfully.
2. Web server responds successfully for http://127.0.0.1:15000/hello and http://[::1]:15000/hello.
3. Telegraf for Windows successfully monitors both URLs.
4. Telegraf for Linux successfully monitors both URLs.

Actual behavior:

1. Web server starts successfully.
2. Web server responds successfully for http://127.0.0.1:15000/hello and http://[::1]:15000/hello.
3. Telegraf for Windows successfully monitors only http://[::1]:15000/hello URL:

http_response,host=win10,method=GET,result=connection_failed,server=http://127.0.0.1:15000/hello result_type="connection_failed",result_code=3i 1606058770000000000
http_response,host=win10,method=GET,result=success,server=http://[::1]:15000/hello,status_code=200 content_length=6i,result_type="success",result_code=0i,response_time=0.0020097,http_response_code=200i 1606058770000000000

4. Telegraf for Linux successfully monitors only http://127.0.0.1:15000/hello URL:

http_response,host=ubuntu,method=GET,result=success,server=http://127.0.0.1:15000/hello,status_code=200 result_code=0i,response_time=0.02016597,http_response_code=200i,content_length=6i,result_type="success" 1606058900000000000
http_response,host=ubuntu,method=GET,result=connection_failed,server=http://[::1]:15000/hello result_type="connection_failed",result_code=3i 1606058900000000000

Additional info:

Why Telegraf for Windows fails for IPv4 URL http://127.0.0.1:15000/hello and Telegraf for Linux fails for IPv6 URL http://[::1]:15000/hello?
It is because in this line

telegraf/plugins/inputs/http_response/http_response.go

Line 209 in b39c197

return &net.TCPAddr{IP: naddr.IP}, nil

plugin chooses first available address for given interface. In my case it is IPv6 for Windows and IPv4 for Linux.

Would it work if any address ordering solution is provided (so the same address would be chosen
independently from OS type)?
No, they would just fail for the same address type. Other address type would still not work. It is because HTTP Client with chosen address is configured per plugin and not per provided URL.

[powersj]

Hi @zak-pawel,

Would this be an issue for anything other than URLs using localhost? If not, I'm wondering if we document this and move on?

Thanks!

[zak-pawel]

@powersj Yes, it would.

Lets assume I have interface ens33 with IPv4: 192.168.0.192 and IPv6: fe80::43ac:7835:471a:faba.
In Linux I can start 2 web servers like this:

IPv4:

	http.HandleFunc("/hello", func(writer http.ResponseWriter, request *http.Request) {
		fmt.Fprintf(writer, "hello ipv4\n")
	})
	http.ListenAndServe("192.168.0.192:15000", nil)

IPv6:

	http.HandleFunc("/hello", func(writer http.ResponseWriter, request *http.Request) {
		fmt.Fprintf(writer, "hello ipv6\n")
	})
	http.ListenAndServe("[fe80::43ac:7835:471a:faba%ens33]:15000", nil)

(%ens33 is for specifying zone index, like described here)

I can curl both addresses:

pzak@ubuntu:~$ curl http://192.168.0.192:15000/hello
hello ipv4
pzak@ubuntu:~$ curl http://[fe80::43ac:7835:471a:faba%25ens33]:15000/hello
hello ipv6

(%25 escapes % sign)

telegraf.conf:

[[inputs.http_response]]
  ## List of urls to query.
  urls = ["http://192.168.0.192:15000/hello", "http://[fe80::43ac:7835:471a:faba%25ens33]:15000/hello"]

  ## Interface to use when dialing an address
  interface = "ens33"

Telegraf output:

./telegraf --config telegraf.conf 
2022-02-11T18:11:05Z I! Starting Telegraf 1.22.0-6b186021
2022-02-11T18:11:05Z I! Loaded inputs: http_response
2022-02-11T18:11:05Z I! Loaded aggregators: 
2022-02-11T18:11:05Z I! Loaded processors: 
2022-02-11T18:11:05Z I! Loaded outputs: file
2022-02-11T18:11:05Z I! Tags enabled: host=ubuntu
2022-02-11T18:11:05Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"ubuntu", Flush Interval:10s
http_response,host=ubuntu,method=GET,result=success,server=http://192.168.0.192:15000/hello,status_code=200 response_time=0.00053921,http_response_code=200i,content_length=11i,result_type="success",result_code=0i 1644603070000000000
http_response,host=ubuntu,method=GET,result=connection_failed,server=http://[fe80::43ac:7835:471a:faba%25ens33]:15000/hello result_type="connection_failed",result_code=3i 1644603070000000000

Same error, same problem (tested on current master branch, post v1.21.3).

I assume that the same problem will exist for Windows for these addresses, but don't have possibility to check right now.

[powersj]

It sounds like the following would need to happen:

1. create a new client during each URL gather or somehow store a client for each URL
2. parse the URL and determine if it is ipv4 and/or ipv6
3. update the interface logic to get an IP on the matching URL's protocol version

Does that sound about right?

[powersj]

@zak-pawel is this something you still want to resolve? If so could you look at the above?

[zak-pawel]

@powersj Sure, I'll propose a solution and open a PR. Sorry, I must have missed your comment from 2022... :/