feat(outputs): Add framework to retry on startup errors

[srebhan]

Summary

A frequently requested feature is to retry startup for plugins in case of error e.g. if the connection to a remote machine fails.

Instead of implementing the logic for retrying in each plugin, this PR provides a framework for output-plugins to use if they wish to support retries. As an example, the Kafka output plugin is converted to use the new framework

More details can be found in the TSD-006 specification.

Checklist

• No AI generated code was used in this PR

Related issues

resolves #14803
resolves #9778

[powersj]

I really like this.

In general, all outputs are connecting to something or creating a server for something to connect to with the exception file? And I don't think we would want to change anything for file.

What do you think are the next steps?

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz

[powersj]

Thank you for this! Did you want to look at the wavefront graphite output in this or as a second PR?

[srebhan]

@powersj here are my findings regarding the graphite output... The current behavior of graphite is to always succeed on startup even though no server can be connected. The plugin will then try to connect to the given servers in each Gather() cycle. If at least one server was connected successfully, the data is written. This is done unconditionally and corresponds to the retry strategy in the new framework with potentially partial startup. Additionally, servers are reconnected if they are found to be unconnected. This is also tried in each Gather() cycle.

Regarding the framework, the conversion of the graphite output has some drawbacks:

1. To keep the current behavior, we would need to set startup_error_behavior to retry by default. This is difficult and can only be done with a special handling in the config code.
2. The "reconnect disconnected servers" behavior currently implemented in the plugin is currently not covered by the framework code.

So given the challenges above and the little gain in terms of code-removal I suggest to not convert the graphite output for now, at least until we also handle the reconnect case in the framework...

[powersj]

@DStrand1 all yours to do a final review.