Routing metrics by plugin

[joakimberglund]

Feature Request

Add the possibility to select which inputs a certain output should use.

Proposal:

[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "DB1"
  inputs = [ "exec" ]

[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "DB2"
  inputs = [ "mem", "cpu" ]

And/or.....

[[outputs.influxdb]]
  name ="output1"
  urls = ["http://localhost:8086"]
  database = "linux"

[[inputs.cpu]]
  output = ["output1", "output2"]
  percpu = false
  totalcpu = true

Default should be that it behaves as it does today.

Current behavior:

Not possible

Desired behavior:

Use case: [Why is this important (helps with prioritizing requests)]

Using the namepass/fieldpass/tagpass is not flexible enough especially when using the exec och httpjson inputs where the name can change or contain multiple names which might collide namewise with other inputs.

Also supporting dividing up the measurements in different databases so that 1) it's easier to scale influxdb instanses 2) get better performance and 3) easier for users to design dashboards

[sparrc]

this would be a nice feature to have, but it's actually quite a tricky problem. Internally all plugins are placed in lists and there is no way to differentiate between two different [[outputs.influxdb]] output instances.

FWIW, the 3 current best workarounds are:

• use name_override, name_suffix, or name_prefix config options to prepend names to your measurements on the input-side, then use namepass to route to outputs, ie:

[[inputs.exec]]
  # ...
  name_prefix = "exec1_"

[[inputs.exec]]
  # ...
  name_prefix = "exec2_"

# this influxdb will only get exec1
[[outputs.influxdb]]
  # ...
  namepass = "exec1_*"

# this will only get exec2
[[outputs.influxdb]]
  # ...
  namepass = "exec2_*"

• Use tags, tagdrop, tagpass routing, see @phemmer's comment below
• run multiple telegraf processes.

[phemmer]

I currently do this in my environment. My solution looks like:

[[outputs.influxdb]]
  urls = ["http://influxdb.example.com"]
  database = "db_default"
  [outputs.influxdb.tagdrop]
    influxdb_database = ["*"]

[[outputs.influxdb]]
  urls = ["http://influxdb.example.com"]
  database = "db_other"
  tagexclude = ["influxdb_database"]
  [ouputs.influxdb.tagpass]
    influxdb_database = ["other"]

[[inputs.foo]]
  [inputs.foo.tags]
    influxdb_database = "other"

This will send inputs.foo to the "db_other" influxdb output. All other inputs (which do not have a influxdb_database tag) will go to the "db_default" influxdb output.

[joakimberglund]

@phemmer Now thats clever, I can use that, thanks

Another way that this might be doable to create almost the same effect is that the configs in telegraf.d is their own instanses.
Aka with "private" input/outputs lists per file.

[daraeburn]

I have tried that method using config files @joakimberglund but can't seem to get it to work. When I run a config test on it once I move my input and output files to /etc/telegraf/telegraf.d/ it fails with the following error

2016/09/21 09:37:25 Error: no outputs found, did you provide a valid config file?

It looks like it isn't reading the files in the telegraf.d folder even after trying to add include "/etc/telegraf/telegraf.d/*.conf" to the end of the file. If I add that it fails with this error :

Error parsing telegraf.conf, toml: line 147: parse error

I'm wondering if there is some kind of test in telegraf that checks for an output section before allowing it to run. I'm going to try adding just one input and output plugin then add the others to the telegraf.d folder to see if that works.

[daraeburn]

That doesnt work either. I added just the default input and output plugins to the config and added in the other input and output plugins in to telegraf.d folder but it only loads the default one and ignores the others.

[sparrc]

@daraeburn: please open a separate issue with steps to reproduce if you are having a configuration problem, I don't think that's related to this issue.

[andig]

I can't get this to work. Since I don't know the metric names I've tried to setup routing influx v1 to a v2 instance based on tags:

[[inputs.influxdb_listener]]
  service_address = ":9986"
  tags = { influx_routing = "volkszaehler" }

[[outputs.influxdb_v2]]
  urls = ["http://influxdb2:9999"]
  organization = "home"
  bucket = "volkszaehler"
  tagpass = { influx_routing = ["volkszaehler"] }
  tagexclude = ["influx_routing"]

[[outputs.influxdb_v2]]
  urls = ["http://influxdb2:9999"]
  organization = "home"
  bucket = "metrics"
  tagdrop = { influx_routing = ["*"] }

Update I had to switch tagdrop and tagexclude around. The option naming is not too obvious, but now it's in line with @phemmer's example above.

[dikkini]

any news?

[sjwang90]

Please leave a comment describing why the example above or the tagpass/tagdrop documentation outlined here doesn't resolve what you need.

Otherwise we will close this issue.

[telegraf-tiger]

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Page. Thank you!