Merging a number of plugins

[srclosson]

Feature Request

Hi Team, over the years, I've developed a number of plugins, and would like to discuss how I can get all of these "up to snuff" and merged into the main code. I expect there will be a little work just to get them all documented properly, but I feel at least submitting a report and starting off with some discussion of order would be helpful.

Proposal:

The current new plugins I've developed are:

Input Plugins

1. dirmon: dirmon recursively monitors a directory (an input directory). It tries to match files as they arrive to a regex and processes matching files according to the associated parser (data_format). Metadata about the file, and file name can be matched using regex's into a "fileinfo" metric. "fileinfo" metrics can then be passed to any output plugin to be stored, or to a "transfer" output plugin to be moved somewhere else (out of the way).
2. nmea gps: The nmea plugin looks to a serial port or tcp port for nmea gps messages and will parse relevant data. Nmea sentences are parsed according to a match, if you want to only match certain ones.
3. ping: This was a rewrite of the current ping tool to not spawn new ping processes. We had a requirement to do high resolution pings, so I rewrote so that we had an actual ICMP packet creator that would send directly without a new process., CPU usage and memory requirements went down heavily.
4. sqlquery: The SQL query plugin queries an SQL server table for data (but could match other sql dialects as well). If you select "column" as tag_column the column data will be read into a tag, rather than a field. I will typically use a strategy of "read every 30 seconds for 30 seconds of data".
5. wmi: The wmi plugin queries wmi directly through COM, for data in any wmi namespace.
6. win_perf_counters: I just made a small enhancement so that I could query remote computers as well as local computers for performance data.

Output Plugins

1. postgresql (timescale): There is already lots of development here, but we had very high write requirements and so I had to optimize this plugin as much as possible so that it would insert and not backup other output plugins. It's quick quick, although I've not tested other plugins.
2. transfer: The transfer plugin looks for metrics that contain information about a file, and will send that file to some other destination. Current output mechanisms are sftp, ftp, and smb shares as well as a local copy.
3. sqlserver: The sql server plugin writes metrics to a microsoft sql server table.
4. rollingfile: A rolling file output plugin writes to temporary files. It's configurable by time, and size. So when your file reaches a certain duration, or size, the file is closed, moved somewhere else, and a new file is started. Good for offloading batches of data.

Processor Plugins

1. ankoscript: Ankoscript uses the anko scripting language to manipulate a metric. The scripting language is quite flexible, and quite "go" like. I will often use it when no other processor plugin does exactly what I want. You can call functions directly on your metric (ex: metric.Tags(), or metric.Timestamp()).
2. geohash: The geohas plugin looks for a configurable latitude and longitude in a group of metrics, and if found, will add an additional field for a geohash.
3. lookup: The lookup plugin is much like the grok processor. It combines the power of the enum processor and the grok processor by allowing the user to match fields in a message, and correlate them, or translate them using a lookup table.
4. utm2latlon: Much like the geohash plugin, utm2laton will take utm coordinates, and either decorate or replace them with a latitude and longitude.

Service Plugins

A service plugin was a new plugin type that I added. It's essentially a stand alone service. Hopefully the plugins for it make sense, as I felt they didn't fit into any other category. A service plugin has a built-in pub-sub messaging bus that enables service plugins to send messages to other service plugins, but the message format is unstructured, and I've not used it yet.

1. influx_bridge: influx_bridge filters influx responses, and will allow you to filter out fields in a response. The grew from the requirement for some systems to only see specific fields in an influx response. For example, perhaps we wanted one system to only see fields for cpu usages etc...
2. influx_cache: Influx cache intercepts influx responses, and will cache them in a leveldb. Responses that are hits, never get sent to the backend database. The use case that spawned this was if multiple users had the same data-intensive dashboard open and all requested the data at nearly the same time, the first request would be cached and delivered, and the remainder of the responses would receive the same cached response, allowing only one request to be send to the backend influx database.

Use case:

I would like to have some place to tackle all these plugins in a somewhat orderly way.
I've written several serializer and parser plugins, which I will update this issue with as well.

[glinton]

Woah, that sounds like a lot of great work! There is active work this week on enhancing the current ping plugin to do essentially what you've described. It'd be great to check out what you've got for that and other plugins. I think what would work well would be to open a pull request for each plugin that we can individually review and merge in as appropriate. I'm sure we can help prioritize them before you spend time opening all those PR's.

[russorat]

@srclosson glad to hear from you! I'd love to see these plugins added!

[danielnelson]

As mentioned, we do need to evaluate these one at a time, and it will take some time to go through the full list. I suggest picking the top one or two you want first and open pull requests with the code. Here are some initial reactions to help you pick which ones might be good candiates to get started.

One thing to keep in mind is even if we don't accept a plugin, there is a future path where they could live outside of Telegraf in a way that wouldn't require you to recompile them for every Telegraf release. There isn't a timeline for this, and honestly we don't know how exactly it will happen (probably grpc), but it is something we want to provide.

Inputs:

1. Sounds similar to the file input, but with some additional collection about the file. Is this fileinfo attached to the data parsed, or are separate metrics synthesized for the meta data?

2. 👍

3. We have some ongoing work with ping going on internally, but it would be nice to see the code you have and maybe we can integrate it.

4. Would like to see the code. There are existing pull requests, Add sqlserver_extensible input plugin #3069 and Genericsqlplugin #2785, but the current plan is to finish Genericsqlplugin #2785 so we have a single input plugin that can handle multiple databases.

5. 👍

6. 👍

Outputs:

1. I'm sure have seen PostgreSQL output plugin #3428, one challenge of writing SQL outputs is the number of ways the data could be structured, without any clear way that is "correct". Could this pull request work for you and if not, could you comment on that issue?

2. It sends the file itself? Interesting, but I think we probably wouldn't accept this plugin.

3. 👍 but comments on clarifying config file #1 apply here as well.

4. There is a pull request for this on the file output Add file rotation based on file age to file output plugin #5547, can you check out the implementation there and comment about if it would be fit for your purposes or why not?

Processors:

1. I don't think we would accept this plugin currently.

2. 👍

3. 👍 but probably should be split into 2 plugins, one for grok and one for the lookup.

4. 👍

Service Plugins:

1 & 2. It's really cool how you were able to use Telegraf successfully for these uses but I think we probably wouldn't add these since they deal with the query side of things.

[sjwang90]

File directory monitoring plugin (dirmon input + transfer output) merged in #8751. Closing issue.

@srclosson If you would like to contribute any of your other plugins, we'd love any PRs 😄