Add support for Docker-machine labels in docker input plugin

[blaggacao]

Feature Request

Opening a feature request kicks off a discussion.

Proposal:

By finally allowing docker daemon labels (also known through docker-machine create command) in [global_tags], in the case of a docker-based cluster (such as rancher.io as for example), where docker daemon lables are extensively used to sort, classify and for cluster scheduling, influx db would get a more native look and feel and benefit from a deeper ecosystem integration.

So far that's without questioning, however: Should it?

Some Pros:

• Docker is widely used
• Docker get's a lot of traction

Some Cons:

• Docker is not the whole and only picture of influx applications
• A special treatment might not be warranted (?)
• Lifting (some) technical integration from the plugin level to the telegraf level

Current behavior:

[[global_tags]]

Desired behavior:

[[global_tags]]
  [[docker.daemon.labels]]
  io.rancher.service.name # Pseudo-example label

Use case:

Ordered by selfishness: 😄
3882 rancher stargazers would be invited to hail to influxdata
https://github.com/rancher/rancher/stargazers
the docker crowd would be pushed to stop using part-stack solutions
https://github.com/docker/docker/stargazers

[sparrc]

interesting, could certainly be a use-case for a new type of "tagger" plugin that could dynamically fetch relevant data for particular measurements.

As a non-expert of docker, can you explain how the "docker daemon labels" differ from "docker container labels"? And what each is used for?

Also, any reason this couldn't just be implemented as a feature of the docker input plugin?

[blaggacao]

The daemon labels are scoped per machine, so for example take the following situation:

• I want to spin up a new machine in sfo3 datacenter with it being scheduled to become a database instance
• I put two labels on it: com.example.region = sfo3 and com.example.role = database
• Probably I also want to give it a name of some funny animal, such as com.example.name = hedgehog

Those a beared by the daemon and now I can tell my scheduler (e.g. rancher) - or more likely I have already told the scheduler before - to deploy the database image on all com.example.role = database, so db image will be spun there. I also have told my scheduler to put a telegraf on every instance, so as soon hedghog is up and running, telegraf will be spun up too.
Now we go: having com.example.name in the global section will make magically appear in my TICK stack logs coming from hedgehog.

I can (!) manually get the same by tinkering with environmental variables and put them in the global section, but from an operator perspective this feels murky, as I'm replicating a single concept onto different transport mechanisms just for compatibility. Terrible!

I am assuming the docker input plugin is scoped to containers, while the information found in the daemon labels has by nature a machine scope. This kind of information, if my interpretation is right, is conceived to belong to the [[global_tags]] directive.

Note for better understanding:

• Docker set to replace your init system
• Docker set to drastically change your way to use your OS
• Docker could probably live on a kernel only, see RancherOS for a very streamlined approach

So the idea of having a traditionally scoped operating system layer is questioned more and more. (see Alpine as Docker Host as in MobyLinux for Windows, RancherOS). This is why congruence of the two concepts "host machine" and "docker daemon" is ever increasing.

[sparrc]

I think you may be mistaken about the docker input plugin. The plugin collects metrics for each container, but also about the overall docker runtime,

for example, even if I'm not running any containers I still get some metrics like this:

% telegraf -config ~/db/ws/telegraf.conf -input-filter docker -test
* Plugin: docker, Collection 1
> docker,host=tyrion n_containers=3i,n_cpus=2i,n_goroutines=28i,n_images=14i,n_listener_events=1i,n_used_file_descriptors=16i 1469637536000000000
> docker,host=tyrion,unit=bytes memory_total=2098126848i 1469637536000000000

the daemon tags could be added to these metrics, and also to the individual container metrics.

[blaggacao]

I was indeed.

the daemon tags could be added to these metrics, and also to the individual container metrics

Yes!

With that, the remainder of this feature request would then be docker to become a very first class citizen in the influx stack, which can be fine under "maybe later" (or never), depending on the actual development focus.

[sparrc]

what do you mean by a "very first class citizen"? I'm happy to prioritize any features that would help using Telegraf to collect docker metrics, I'm just not sure what they are :)

[blaggacao]

Sorry, this was a little too implicit: I was referring to the [[global_tags]] section integration.

So reformulating the picture my request would look like this:

• try to integrate with the docker labels (daemon and containers)
• if possible put the daemon labels in a shortcut to the globals section

[misko321]

the daemon tags could be added to these metrics, and also to the individual container metrics.

This would be a very useful feature, adding tags from the docker daemon itself the same way that container labels are added as tags. Currently, there is no way to browse containers (in Grafana for instance) based on the docker daemon tags, but only through container tags. Because of that we need to duplicate labels from docker daemon to each and every container to make them appear in Grafana.

The [[global_tags]] stuff in not a requirement for me, I would just want to have those tags included.

[reimda]

Since so much in telegraf has changed since this discussion, including that labels are optionally included as tags (see #2425) I'm going to close this. If you still need a specific docker change, please open a new issue. Thanks!