Setting 'tag_keys' dynamically

[gaddevijay]

In the work that I am doing, I do not know in advance the tags to be used in tag_keys. For every packet that is received I can look at the data and decide what the tags could be. Is there any way that I can set the 'tag_keys' per record that I am will be sending to influxDB?

For example, I could get a packet which could transform to {'interface-name': 'abc', 'interface-bandwidth': 1234, }. For this the tag should be 'interface-name'. Next I could get a packet which could transform to {'arp-ip': '1.2.3.4' and 'arp-timeout': 34}. Here the index should be 'arp-ip'. All of the packets are coming at the same time, so the timestamp would be the same and these should of to the same measurement. Above records are just example to simplify the use case.

[saidie]

👍

I would like to dynamically set tags, too. It would be possible by combining rewrite_tag_filter and forest plugins but it is tough. The auto_tags option can work sometimes but I feel it is hard-to-use because I need to carefully choose fields and to check a type of filed is not changed by passing a record thorough some plugins.

A simple solution would be adding an option like "tag_key" (confusing name though...) and then post the value of the "tag_key" field as tags.
For example, if tag_key is set to tags, you would be able to send records like:

{ 'interface-bandwidth': 1234, 'tags': { 'interface-name': 'abc' } }
{ 'arp-timeout': 34, 'tags': { 'arp-ip': '1.2.3.4' } }

[repeatedly]

yeah, saidie's way is one approach.
Fluentd can't judge which field is tag or not because Fluentd is not human.
So need parameter or rule for dynamic tags.

[gaddevijay]

Or if we send like below, can we use the 'tag_keys' in the message?

{ 'interface-bandwidth': 1234, 'tag_keys': [ 'interface-name' ] }
{ 'arp-timeout': 34, 'tag keys': [ 'arp-ip'] }

The code modification is minimum and has backward compatibility as well.
Downside is 'tag_keys' will become a reserved key.

[CheyiLin]

👍 for @gaddevijay 's proposal.
An user defined tag_keys field contains a list of fields need to be tagged for current record, is pretty practical.

[repeatedly]

Maybe, adding tag_keys_field parameter is better for me.