Implement Elasticsearch plugin

[brocaar]

This implements a plugin for getting the stats from Elasticsearch.
See also: https://www.elastic.co/guide/en/elasticsearch/reference/1.6/cluster-nodes-stats.html

[alvaromorales]

👍 nice. I was working on something similar ... you beat me to it :-)

I had a couple of questions about your design.

1. Why did you decide to use query the Elasticsearch API directly instead of using a Go client?

elastigo, for example, has nice connection functionality. It keeps a pool of ES hosts, runs periodic health checks, and uses the Epsilon Greedy algorithm to mark failed nodes.

I'm not sure what's best practice for Telegraf. Some plugins (e.g., kafka, rethinkdb) use clients and others just query a service directly (e.g. memcached, redis).

2. Your plugin only gets the indices stats. Is this temporary, or do you envision other Elasticsearch stats (os, jvm, process, etc) being separate plugins?

This plugin might be a good use-case for pass/drop plugin options. Since measurements are prefixed with stat type, if a user only cares about indices stats, they can set pass = ["indices"].

[alvaromorales]

node_name should be added to tags. node_uuid perhaps, but I'm not sure if it would be very useful.

[brocaar]

• Will do

[otoolep]

+1. Please do make good use of tags. Node name, hostname if possible, etc.

On Wednesday, July 8, 2015, Alvaro Morales notifications@github.com wrote:

In plugins/elasticsearch/elasticsearch.go
#55 (comment):

•   return err
  

• }
• if r.StatusCode != http.StatusOK {

•   return fmt.Errorf("elasticsearch: API responded with status-code %d, expected %d", r.StatusCode, http.StatusOK)
  

• }
• d := json.NewDecoder(r.Body)
• esRes := &struct {

•   ClusterName string           `json:"cluster_name"`
  

•   Nodes       map[string]*node `json:"nodes"`
  

• }{}
• if err = d.Decode(esRes); err != nil {

•   return err
  

• }

• for _, n := range esRes.Nodes {

•   tags := map[string]string{
  

node_name should be added to tags. node_uuid perhaps, but I'm not sure if
it would be very useful.

—
Reply to this email directly or view it on GitHub
https://github.com/influxdb/telegraf/pull/55/files#r34196293.

[brocaar]

Will add the following too to the tags:

• Node UUID
• Node properties

[brocaar]

To address your question about not using a client, I think that for an application like Telegraf it is better to keep the number of dependencies as low as possible. In the end, all the libraries (as far as I know) wrap the Elasticsearch REST API. I've implemented the plugin in such a way that it can run on every node within a Elasticsearch cluster (since you might be as well interested in the machine load, disk space, mem usage etc...), or if you prefer it will fetch the stats from all the nodes within the cluster (local = false). I'm not sure which advantage a library has over consuming the API directly. Probably these libraries will perform healthchecks by making API calls too :-)

To be honest, I'm not 100% sure if all the other stats are relevant, but I think you're right and some will be. This shouldn't be a problem however, since I'm prefixing all stats with indices, so there won't be any collision. But I'm up for looking at the other metrics too (maybe this pull-request, maybe in a separate one?)

[brocaar]

I've updated the tags. Node ID and the node attributes are now added to the tags as well. Will look into implementing the other stats.

[brocaar]

@otoolep @alvaromorales I've updated my pull-request. Please let me know what you think!

[alvaromorales]

minor typo: transport

[alvaromorales]

Looks really good!

[brocaar]

Thanks for the feedback. I fixed the typo :-)