custom tags are not per input instance

[lucadistefano]

Feature Request

Assign a custom tag for each input plugin instance.
Actually you can add custom tags only per plugin class

For instance in the case of win_perf_counters could be very useful because the ObjectName often contains the application instance name and this info should be traced in a separate tag (as per influxdb guidelines)
Example:
\SQLServer$PROD:Memory Manager\Total Server Memory (KB)

	ObjectName = "SQLServer$PROD:Memory Manager"
	Counters = ["Total Server Memory (KB)"]
	Instances = ["-----"]
	Measurement = "sqlserver"
	[inputs.win_perf_counters.object.tags]
	      application_instance = "PROD"
		parent_instance = "SQLServer"

SQLServer
actually in influx the tag added by the plugin is objectname="SQLServer$PROD:Memory Manager" but it is a compund of 3 infos: the application instance the 'parent' and the performance counter objectname

The custom tag per plugin instance could be a useful mechanism for all plugins.

Relevant telegraf.conf:

[[inputs.win_perf_counters]]
	[[inputs.win_perf_counters.object]]
		ObjectName = "Process"
		Counters = ["Handle Count"]
		Instances = ["telegraf"]
		Measurement = "win_proc"

	[[inputs.win_perf_counters.object]]
		ObjectName = "Process"
		Counters = ["Handle Count"]
		Instances = ["nsi"]
		Measurement = "win_proc"
		  
	[[inputs.win_perf_counters.object]]
		ObjectName = "Process"
		Counters = ["Handle Count"]
		Instances = ["csrss"]
		Measurement = "win_proc0"
	 [inputs.win_perf_counters.object.tags]
		  application_instance = "ZZZ"
	  
	[[inputs.win_perf_counters.object]]
	   ObjectName = "Process"
	   Counters = ["IO Data Operations/sec"]
	   Instances = ["_Total"]
	   Measurement = "win_proc1"
      
	[[inputs.win_perf_counters.object]]
	   ObjectName = "Process"
	   Counters = ["IO Data Operations/sec"]
	   Instances = ["*"]
	   Measurement = "win_proc2"

System info:

windows server 2012R2
telegraf win 64 bit 1.2.1

Steps to reproduce:

use the inputs of above for put data in influxdb

Expected behavior:

only data for process instance csrss in measurement win_proc0 should have additional tag application_instance = "ZZZ"

Actual behavior:

all metrics of plugin inputs.win_perf_counters in all measurements contains additional tag
application_instance = "ZZZ"

Thanks

[danielnelson]

Would it be possible to have one [[inputs.win_perf_counters]] per object, each with its own tags table? I imagine this might have performance issues even if it works in the same way.

[lucadistefano]

if I configure telegraf in this way:

[[inputs.win_perf_counters]]
	[[inputs.win_perf_counters.object]]
		ObjectName = "Process"
		Counters = ["Handle Count"]
		Instances = ["VMTools"]
		Measurement = "win_proc"
	[[inputs.win_perf_counters.tags]]
		  application_instance = "XXX"		
[[inputs.win_perf_counters]]
	[[inputs.win_perf_counters.object]]
		ObjectName = "Process"
		Counters = ["Handle Count"]
		Instances = ["Notepad++"]
		Measurement = "win_proc0"
	   [[inputs.win_perf_counters.tags]]
		  application_instance = "ZZZ"
		  
[[inputs.win_perf_counters]]
	[[inputs.win_perf_counters.object]]
		ObjectName = "Process"
		Counters = ["Handle Count"]
		Instances = ["telegraf"]
		Measurement = "win_proc1"

then only the last definition of win_perf_counters is polled: only win_proc1 will be filled.
bug?

[danielnelson]

It might be, can you post a small sample of the output, perhaps using

[[processors.printer]]

[lucadistefano]

First case only last measurement is written:

[global_tags]
# Configuration for telegraf agent
[agent]
  interval = "5s"
  round_interval = true
  metric_buffer_limit = 1000
  flush_buffer_when_full = true
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  debug = true
  quiet = false
  #logfile = "/Program Files/Telegraf/telegraf.log"
  ## Override default hostname, if empty use os.Hostname()
  hostname = ""

###############################################################################
#                                  OUTPUTS                                    #
###############################################################################

# Configuration for influxdb server to send metrics to
[[outputs.influxdb]]
  urls = ["http://nbox-demo.wp.lan:8086"] # required
  database = "test" # required
  precision = "s"
  timeout = "5s"

[[processors.printer]]

###############################################################################
#                                  INPUTS                                     #
###############################################################################

[[inputs.win_perf_counters]]
	PrintValid=true
	[[inputs.win_perf_counters.object]]
		ObjectName = "Processor"
		Counters = ["% User Time"]
		Instances = ["_Total"]
		Measurement = "agent"
		#IncludeTotal=false #Set to true to include _Total instance when querying for all (*).
	   [[inputs.win_perf_counters.tags]]
		  application_instance = "AAA"

[[inputs.win_perf_counters]]
	PrintValid=true
	[[inputs.win_perf_counters.object]]
		ObjectName = "Process"
		Counters = ["Handle Count"]
		Instances = ["spoolsv"]
		Measurement = "win_proc1"
	   [[inputs.win_perf_counters.tags]]
		  application_instance = "BBB"
		
[[inputs.win_perf_counters]]
	PrintValid=true
	[[inputs.win_perf_counters.object]]
	   ObjectName = "Process"
	   Counters = ["IO Data Operations/sec"]
	   Instances = ["_Total"]
	   Measurement = "win_proc2"
	   [[inputs.win_perf_counters.tags]]
		  application_instance = "CCC"
		

C:\Program Files\Telegraf>telegraf.exe --debug --config test.conf
2017-03-21T13:28:24Z D! Attempting connection to output: influxdb
2017-03-21T13:28:24Z D! Successfully connected to output: influxdb
2017-03-21T13:28:24Z I! Starting Telegraf (version 1.2.1)
2017-03-21T13:28:24Z I! Loaded outputs: influxdb
2017-03-21T13:28:24Z I! Loaded inputs: inputs.win_perf_counters inputs.win_perf_counters inputs.win_perf_counters
2017-03-21T13:28:24Z I! Tags enabled: host=WIN-MONTEST
2017-03-21T13:28:24Z I! Agent Config: Interval:5s, Quiet:false, Hostname:"WIN-MONTEST", Flush Interval:10s
Valid: \Process(_Total)\IO Data Operations/sec
win_proc2,instance=_Total,objectname=Process,host=WIN-MONTEST IO_Data_Operations_persec=0 1490102910000000000

win_proc2,instance=_Total,objectname=Process,host=WIN-MONTEST IO_Data_Operations_persec=0 1490102910000000000

win_proc2,instance=_Total,objectname=Process,host=WIN-MONTEST IO_Data_Operations_persec=0 1490102910000000000

win_proc2,instance=_Total,objectname=Process,host=WIN-MONTEST IO_Data_Operations_persec=0 1490102915000000000

win_proc2,instance=_Total,objectname=Process,host=WIN-MONTEST IO_Data_Operations_persec=0 1490102915000000000

win_proc2,instance=_Total,objectname=Process,host=WIN-MONTEST IO_Data_Operations_persec=0 1490102915000000000

2017-03-21T13:28:35Z D! Output [influxdb] buffer fullness: 6 / 1000 metrics.
2017-03-21T13:28:35Z D! Output [influxdb] wrote batch of 6 metrics in 13.03ms

[Dark0096]

+1

[danielnelson]

So it's always the last defined plugin that works? Also the tag is still not added. Seems like a bug.

[nheath]

This is a blocking issue for us, also with database perf counters. Since we have multiple instances, we can only monitor one of them. I'd tried declaring multiple [[inputs.win_perf_counters]] as was suggested in the similar #2572, but this didnt work for perf counters.

[danielnelson]

I'm going to close this as a duplicate of #1137