Detect changes to config and reload telegraf (copy of pr #8529)

[sspaink]

resolves #7187

This feature was thought to be already supported, but has now made clear that it isn't and is required for another project effort. To help get this pull request ready quicker I've taken it over to resolve merge conflicts and use the latest influxdata/tail now that the fix has been merged. This pr still needs to be tested and reviewed to make sure this is the best approach. I also didn't copy over the windows specific changes because it wasn't obvious why they were necessary, see the original pull request #8529 for more details.

Lots of kudos to @vlastahajek for the original pr and the fix to the tail repo.

[telegraf-tiger]

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

• darwin_amd64.tar.gz

• windows_amd64.zip

• windows_i386.zip

• aarch64.rpm

• linux_arm64.tar.gz

• arm64.deb

• s390x.rpm

• linux_s390x.tar.gz

• s390x.deb

• linux_mips.tar.gz

• mips.deb

• ppc64le.rpm

• linux_ppc64le.tar.gz

• ppc64el.deb

• armel.rpm

• linux_armel.tar.gz

• armel.deb

• armv6hl.rpm

• freebsd_armv7.tar.gz

• linux_armhf.tar.gz

• armhf.deb

• linux_mipsel.tar.gz

• mipsel.deb

• x86_64.rpm

• freebsd_amd64.tar.gz

• linux_amd64.tar.gz

• amd64.deb

• i386.rpm

• freebsd_i386.tar.gz

• linux_i386.tar.gz

• i386.deb

• static_linux_amd64.tar.gz

[sspaink]

@wojciechka, @ssoroka Are you able to take the artifacts posted by the Telegraf Tiger and see if this works for you? I am not entirely sure what the best way to test it would be. If you can provide some steps on how to test it, I'd be happy to do it as well.

[wojciechka]

@wojciechka, @ssoroka Are you able to take the artifacts posted by the Telegraf Tiger and see if this works for you? I am not entirely sure what the best way to test it would be. If you can provide some steps on how to test it, I'd be happy to do it as well.

Unfortunately this does not work inside Kubernetes pods that have their configurations mounted as secrets.

This is because the secrets are handled as symlinks to symlinks and this logic may not be detecting all of the underlying changes.

I can confirm that just using --watch-config poll or --watch-config inotify works in the simple case of this being a file, but not when this is a Kubernetes secret.

I have tried to reproduce it outside of Kubernetes, but could not reproduce the case where it does not work, it may be that the mounts are handled by Kubernetes.

Would it be possible to implement it by periodically reading the config file and comparing its contents and/or checksum? This could be something like --compare-config 1m to indicate the check should be made every minute.

[wojciechka]

After spending more time and testing both modes, I have verified that --watch-config inotify is working well with telegraf-operator - where telegraf is run inside Kubernetes and has its config mounted as a Kubernetes secret.

I have verified this in kind and a GKE cluster, so I am optimistic that it will run in any Kubernetes cluster.

[pszlazak]

Is there any plan to support remote config (over http)? If I try this option I get:

2021-12-07T14:40:53Z W! Cannot watch config http://xx.xx.xx.xx:8086/api/v2/telegrafs/aaaaaaaa: stat http://xx.xx.xx.xx:8086/api/v2/telegrafs/aaaaaaaa: no such file or directory