Add configurable Max TTL duration for statsd input plugin entries

[ivorybilled]

The new config value will establish how long any metric can sit in the gauges/counters/timings/sets without being updated / received. When the Max TTL elapses, the metric will stop being reported until it was received again.

closes #8348

[ssoroka]

Nevermind, I got confused about the number of Gather calls. :D it's all good.

[LINKIWI]

After upgrading from Telegraf 1.16.3 to 1.17.0, I've noticed that tags from statsd emissions are being mangled with unrelated metrics. I'm not pointing to this PR as the root cause for the regression, but this is also the only thing that's changed with the statsd plugin since the last release. If I have some more time I'll compile a more thorough bug report with a reproducible example.

[ivorybilled]

Interesting. Thank you for reporting that, let me know when/if you get more info!

[gnjack]

We've also seen the same mangled tags from statsd metrics after upgrading to 1.17.0

[gnjack]

My golang is very rusty, but I think this is the cause of the issue:

The s.Lock() was moved from the start of parseStatsdLine to the start of aggregate. This means the lock has been moved from before tag calculation for a metric to after the tag calculation.

As the parser was recently multithreaded, this means there may be multiple goroutines calling the following line in parseStatsdLine which is now no longer inside a lock:

telegraf/plugins/inputs/statsd/statsd.go

Lines 648 to 649 in 76c2201

	// Parse the name & tags from bucket
	m.name, m.field, m.tags = s.parseName(m.bucket)

func (s *Statsd) parseName(bucket string) does not appear to be thread safe. First it pulls out tags for the current metric it was invoked with

telegraf/plugins/inputs/statsd/statsd.go

Lines 691 to 696 in 76c2201

	// Parse out any tags in the bucket
	if len(bucketparts) > 1 {
	for _, btag := range bucketparts[1:] {
	k, v := parseKeyValue(btag)
	if k != "" {
	tags[k] = v

then creates a GraphiteParser on the shared Statsd struct s

telegraf/plugins/inputs/statsd/statsd.go

Lines 707 to 710 in 76c2201

	if p == nil || s.graphiteParser.Separator != s.MetricSeparator {
	p, err = graphite.NewGraphiteParser(s.MetricSeparator, s.Templates, nil)
	s.graphiteParser = p
	}

and sets the default tags on that shared GraphiteParser to the current metrics tags

telegraf/plugins/inputs/statsd/statsd.go

Line 713 in 76c2201

p.DefaultTags = tags

this means multiple goroutines will be concurrently modifying s.graphiteParser.DefaultTags to different values and calling name, tags, field, _ = s.graphiteParser.ApplyTemplate(name) which will return incorrect tags if another goroutine has modified s.graphiteParser.DefaultTags