Generic handling to ignore errors on startup

[powersj]

Use Case

In numerous feature requests, users are asking for a way to ignore errors on start up. It would be nice to have a high-level generic method to add this to input plugins.

Expected behavior

Agent level setting to set this behavior for all input plugins.

Actual behavior

Currently users will see Telegraf throw an error during start up if an input plugin cannot start. There are a couple exceptions to this where a configuration option is present to avoid these:

  ## Behavior when we fail to connect to the endpoint on initialization. Valid options are:
  ##     "error": throw an error and exits Telegraf
  ##     "ignore": ignore this plugin if errors are encountered
  # connect_fail_behavior = "error"

Additional info

No response

[zak-pawel]

I'm not sure if it necessarily has to be connect.
Sometimes it can be a local file (regular or socket) exposed by some application/service that hasn't started yet. Such situations may occur, for example, right after the OS starts when Telegraf starts before another application/service.

[srebhan]

We agreed on

  ## Behavior when starting the plugin fails e.g. due to connectivity issues. Valid options are:
  ##     "error": exit Telegraf with an error (current behavior)
  ##     "ignore": remove this plugin from further processing if errors are encountered
  ##     "retry": retry starting up the plugin in each gather or flush cycle
  # startup_error_behavior = "error"

[miken32]

Is it safe to assume the linked PR #14884 will apply to input plugins as well as output? I'm thinking specifically of #12959 here, where we're in the absurd situation of losing reporting for dozens of hosts if one of them goes offline.

This issue specifically mentions input plugins, but the PR only says output plugins, which is why I'm looking for clarification.

[srebhan]

@miken32 the target is to provide the functionality for both inputs and outputs. The first PR will tackle the output side (more specifically the framework itself plus one example plugin) but additional PRs will eventually also implement startup-retry framework for input plugins.

Regarding the mentioned issue #12959: This will not be solved by the startup-retry framework alone as it seems like the auth-information is gathered only once in Init() which is wrong as this has to be in Start() to benefit from the framework. In any case, your hosts will still be missing as long as the authentication server is unreachable!

[srebhan]

Reopening as we still need to do the input side...