feat(agent): Add option to skip re-running processors after aggregators

[powersj]

Summary

By default, after an aggregator is run the processors are run again. This can be helpful for users of aggregators to apply similar rules, however, this can also be very, very unexpected. As this behavior can result in aggregated values from getting scaled or computed against a second time.

While users can use the metric selectors to possibly omit the second run, it is nice to have a flag to remove this behavior entirely when it is not needed.

This introduces the skip_processors_after_aggregators config option, false by default.

Using a config with a processor that multiplies a float field by 10 each time it is run, by default:

❯ ./telegraf --config config.toml --test
2024-02-22T16:56:48Z I! Loading config: config.toml
2024-02-22T16:56:48Z I! Starting Telegraf 1.30.0-c4069a4b brought to you by InfluxData the makers of InfluxDB
2024-02-22T16:56:48Z I! Available plugins: 241 inputs, 9 aggregators, 30 processors, 24 parsers, 61 outputs, 6 secret-stores
2024-02-22T16:56:48Z I! Loaded inputs: exec
2024-02-22T16:56:48Z I! Loaded aggregators: minmax
2024-02-22T16:56:48Z I! Loaded processors: starlark
2024-02-22T16:56:48Z I! Loaded secretstores: 
2024-02-22T16:56:48Z W! Outputs are not used in testing mode!
2024-02-22T16:56:48Z I! Tags enabled: host=ryzen
2024-02-22T16:56:48Z D! [agent] Initializing plugins
2024-02-22T16:56:48Z D! [agent] Starting service inputs
2024-02-22T16:56:48Z D! [aggregators.minmax] Updated aggregation range [2024-02-22 09:56:30 -0700 MST, 2024-02-22 09:57:00 -0700 MST]
2024-02-22T16:56:48Z D! [agent] Stopping service inputs
2024-02-22T16:56:48Z D! [agent] Input channel closed
2024-02-22T16:56:48Z D! [agent] Processor channel closed
2024-02-22T16:56:48Z D! [aggregators.minmax] Updated aggregation range [2024-02-22 09:57:00 -0700 MST, 2024-02-22 09:57:30 -0700 MST]
2024-02-22T16:56:48Z D! [agent] Aggregator channel closed
> metric,host=ryzen value=420 1708621009000000000
2024-02-22T16:56:48Z D! [agent] Processor channel closed
2024-02-22T16:56:48Z D! [agent] Stopped Successfully
> metric,host=ryzen value_max=4200,value_min=4200 1708621009000000000

When set to true, note how the value is no longer scaled again.

❯ ./telegraf --config config.toml --test
2024-02-22T16:56:43Z I! Loading config: config.toml
2024-02-22T16:56:43Z I! Starting Telegraf 1.30.0-c4069a4b brought to you by InfluxData the makers of InfluxDB
2024-02-22T16:56:43Z I! Available plugins: 241 inputs, 9 aggregators, 30 processors, 24 parsers, 61 outputs, 6 secret-stores
2024-02-22T16:56:43Z I! Loaded inputs: exec
2024-02-22T16:56:43Z I! Loaded aggregators: minmax
2024-02-22T16:56:43Z I! Loaded processors: starlark
2024-02-22T16:56:43Z I! Loaded secretstores: 
2024-02-22T16:56:43Z W! Outputs are not used in testing mode!
2024-02-22T16:56:43Z I! Tags enabled: host=ryzen
2024-02-22T16:56:43Z D! [agent] Initializing plugins
2024-02-22T16:56:43Z D! [agent] Starting service inputs
2024-02-22T16:56:43Z D! [aggregators.minmax] Updated aggregation range [2024-02-22 09:56:30 -0700 MST, 2024-02-22 09:57:00 -0700 MST]
2024-02-22T16:56:43Z D! [agent] Stopping service inputs
2024-02-22T16:56:43Z D! [agent] Input channel closed
2024-02-22T16:56:43Z D! [agent] Processor channel closed
2024-02-22T16:56:43Z D! [aggregators.minmax] Updated aggregation range [2024-02-22 09:57:00 -0700 MST, 2024-02-22 09:57:30 -0700 MST]
2024-02-22T16:56:43Z D! [agent] Aggregator channel closed
2024-02-22T16:56:43Z D! [agent] Stopped Successfully
> metric,host=ryzen value=420 1708621004000000000
> metric,host=ryzen value_max=420,value_min=420 1708621004000000000

Checklist

• No AI generated code was used in this PR

Related issues

fixes: #7993

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz

[srebhan]

Awesome @powersj! Can you please also check that invalid config options of processors are handled correctly, i.e. they produce an error!? I remember that there is some special handling for those as they are parsed twice...

[powersj]

Awesome @powersj! Can you please also check that invalid config options of processors are handled correctly, i.e. they produce an error!? I remember that there is some special handling for those as they are parsed twice...

Do you off-hand recall what might have existing tests for parsing twice (e.g. agent? config? procesosrs themselves)?

[srebhan]

Looked it up and it seems this is covered by the tests. Sorry for the noise!