can not find docker metrics in influxdb, anyone can help?

[asdfsx]

I start telegraf with the following config

$ more /etc/telegraf/telegraf.d/docker.conf
[[inputs.docker]]
  # Docker Endpoint
  #   To use TCP, set endpoint = "tcp://[ip]:[port]"
  #   To use environment variables (ie, docker-machine), set endpoint = "ENV"
  endpoint = "unix:///var/run/docker.sock"
  # Only collect metrics for these containers, collect all if empty
  container_names = []

and start telegraf by following command:

/usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d  

but can not find any measurements about docker in influxdb:

> show measurements;
name: measurements
------------------
name
cpu
disk
diskio
mem
swap
system

actually, I can see docker datas collected by telegraf

$/usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d  -input-filter docker -test
* Plugin: docker, Collection 1
......
> docker_cpu,com.docker.compose.config-hash=2db93f17fb0fdbb2b3be408209d18ac7eb9f44d787af2df58b6f6601771763cf,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=grafana,com.docker.compose.service=grafana,com.docker.compose.version=1.5.1,cont_id=528cfa640ba2863df3febd0cd28b173527599b8c2d81a26c6965fc3b13b0ea2d,cont_image=grafana/grafana,cont_name=grafana_grafana_1,cpu=cpu1 usage_total=8040078470i 1454561313911620608
......

[asdfsx]

I finally fix this problem~~~~

[sparrc]

@asdfsx what was the issue/solution?

[asdfsx]

@sparrc nothing but restart telegraf several times, then it appeared in influxdb......but I think there is still some problems remain.
Right now I'm trying to use grafana to display the metrics, but I can only see one point on the graph, not a line.

I'm confused by this.....

and when I execute sql:

SELECT mean(usage_total) FROM docker_cpu WHERE time > now() - 1h AND host = 'mesos36' GROUP BY time(15m)

I only get one record

[asdfsx]

It's so strange that, when I start telegraf by exec the command:

/usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d -debug 

I can get docker data in influxdb.
Then I create a graph in grafana, by using sql like below

SELECT mean("usage_total") FROM "docker_cpu" WHERE "host" = 'mesos36' AND $timeFilter GROUP BY time($interval) fill(null)

I get a graph like below

It seems docker_cpu.usage_total is continue growing~
Seems not right.

And when I start telegraf by using systemctl start telegraf, it seems nothing about docker send to the influxdb~~~

SELECT mean(usage_total) FROM docker_cpu WHERE host = 'mesos36' AND time > now() - 1h GROUP BY time(5m)

docker_cpu
time    mean
2016-02-04T07:25:00Z    
2016-02-04T07:30:00Z    
2016-02-04T07:35:00Z    
2016-02-04T07:40:00Z    9535640409061.729
2016-02-04T07:45:00Z    
2016-02-04T07:50:00Z    
2016-02-04T07:55:00Z    9541001545054.232
2016-02-04T08:00:00Z    9542674282644.1
2016-02-04T08:05:00Z    9544248412986.191
2016-02-04T08:10:00Z    
2016-02-04T08:15:00Z    
2016-02-04T08:20:00Z    
2016-02-04T08:25:00Z

[sparrc]

might be a permissions issue, try

sudo -u telegraf /usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d -debug 

The upward slope is normal. Docker's CPU "usage" metric is actually just a counter of CPU ticks used.

[asdfsx]

I think you are right!
I just notice that the telegraf is running under telegraf account.
So I modify the /etc/systemd/system/telegraf.service

[Service]
EnvironmentFile=-/etc/default/telegraf
#User=telegraf
User=root
ExecStart=/usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d ${TELEGRAF_OPTS}
Restart=on-failure
KillMode=process

You can see the telegraf was started by user telegraf before
And now I can see the new data!

BTW. You said that Docker's CPU "usage" metric is actually just a counter of CPU ticks used.
Does that mean I don't need to use mean function?
Just query data like this:

SELECT "usage_total" FROM "docker_cpu" WHERE "host" = 'mesos36'

or I need some function else, like count,sum?

[sparrc]

yes, that query would be fine

[acherunilam]

sudo su telegraf -c '/usr/bin/telegraf -config telegraf.conf -test -filter docker'

works fine. However, the service fails to send the docker metrics and the log fills with multiple instances of

Error getting docker stats: io: read/write on closed pipe

The default permission on the unix socket is 660 (UID:root, GID:docker), and I've added user telegraf to the docker group as well. @sparrc Any idea what's going wrong?

[zstyblik]

@sparrc I'm seeing the same issue with v0.10.3-1:

Feb 23 09:42:21 dev112-12 docker[879]: 2016/02/23 08:42:21 Error getting docker stats: io: read/write on closed pipe
Feb 23 09:42:25 dev112-12 docker[879]: 2016/02/23 08:42:25 Error getting docker stats: read unix @->/var/run/docker.sock: i/o timeout
Feb 23 09:42:25 dev112-12 docker[879]: 2016/02/23 08:42:25 Error getting docker stats: read unix @->/var/run/docker.sock: i/o timeout

However, % docker ps; works just fine.

[zstyblik]

I think this is most likely related to Docker Version as I'm not seeing it at hosts with Docker version 1.8.3, build f4bf5c7, but I'm seeing it at Docker version 1.9.1, build 4419fdb-dirty.

[zstyblik]

I've written small app which is scaled down plugin from Telegraf and I'm getting the same error/seeing the same issue even at Host with Docker v1.8.3. However, I can see "requests" being made in logs:

Feb 23 11:27:30 builder docker[10341]: time="2016-02-23T11:27:30.054869742+01:00" level=info msg="GET /containers/json"
Feb 23 11:27:30 builder docker[10341]: time="2016-02-23T11:27:30.056860514+01:00" level=info msg="GET /containers/6b864d4d17e370abeff82dc0bb6553905f161fc2ec3b8b2e5998ee9bd637f166/stats?stream=false"
Feb 23 11:27:30 builder docker[10341]: time="2016-02-23T11:27:30.057260630+01:00" level=info msg="GET /containers/616487a45616594a2ca671bd0a6f5691cd71fc2c7eee7dfd85cd6f4d6949e0f1/stats?stream=false"
Feb 23 11:27:30 builder docker[10341]: time="2016-02-23T11:27:30.057576478+01:00" level=info msg="GET /containers/6242042c2f252ab5225f0173090cf37dedda8c18cf2de5f28ed52ce57c22d69c/stats?stream=false"
Feb 23 11:27:30 builder docker[10341]: time="2016-02-23T11:27:30.058421662+01:00" level=info msg="GET /containers/c618e64f04f5d9920119b70477d38d76b4761a1c2a8a92ce704e024d231c4dd1/stats?stream=false"

Origin of the message is https://github.com/influxdata/telegraf/blob/master/plugins/inputs/docker/docker.go#L119

I have no idea how to fix this issue, though. I have found no way to increase timeout value or anything related to such setting. It's possible, though, I haven't dug deep enough.

[tripledes]

@zstyblik the docker plugin has a hardcoded timeout of 5s, https://github.com/influxdata/telegraf/blob/master/plugins/inputs/docker/docker.go#L108-L114 which I believe should be more than enough.

I think that should be a configuration setting, but that's a topic for another discussion.

[zstyblik]

@tripledes unfortunately, this setting isn't related to the issue.

[tripledes]

@zstyblik obviously, a closed pipe has nothing to do with a timeout, but you suggested to increase the timeout and I just provided information regarding it being hardcoded ❔