Grouping by App (marathon)

[samek]

Hi,
now that i have cadvisor up and running through marathon I'm getting data into the influxdb.

The problem that I'm facing is that I don't know which app is under docker name.

Let me try to explain:
When marathon starts a docker it gives him a name for example

for cadvisor it's named mesos-3deefa59-6981-4069-8c74-911aead8b396

As you can see in the screenshot I have 2 nginx images also started and they have completely different names.
In the influxdb I cannot differentiate/group by apps since they are presented as mesos name.

Now, Marathon passes some environment variables into the started dockers which could be used to group it.

"MARATHON_APP_VERSION=2015-02-27T13:38:50.135Z",
"HOST=10.0.0.193",
"MESOS_TASK_ID=nginx2.f726a4c5-be85-11e4-82c2-56847afe9799",
"PORT=31005",
"PORTS=31005",
"PORT_80=31005",
"MARATHON_APP_ID=/nginx2",
"PORT0=31005",
"MESOS_SANDBOX=/mnt/mesos/sandbox",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"NGINX_VERSION=1.7.10-1~wheezy"

those filelds are located in the docker dir config.json and should be accessible by cadvisor.

Question is would it be wise to also sent MARATHON_APP_ID into the influxdb in order to group stats by app ?

Or How would this be approached correctly ?

[vmarmol]

Today we have a concept of "namespaces" for container names and those come with a list of aliases used to address the container. Docker containers are under the "docker" namespace and its aliases are the Docker ID and name. This feature may be served by the "marathon" namespace and have the app ID as one of the aliases. This we will then push to the InfluxDB backend.

We probably don't have the bandwidth to tackle this anytime soon, but we will gladly take PRs towards that goal if you'd like :)

[rjnagal]

If we did a generic change of exporting aliases to storage backend,
wouldn't it resolve this problem too?

On Fri, Feb 27, 2015 at 8:51 AM, Victor Marmol notifications@github.com
wrote:

Today we have a concept of "namespaces" for container names and those come
with a list of aliases used to address the container. Docker containers are
under the "docker" namespace and its aliases are the Docker ID and name.
This feature may be served by the "marathon" namespace and have the app ID
as one of the aliases. This we will then push to the InfluxDB backend.

We probably don't have the bandwidth to tackle this anytime soon, but we
will gladly take PRs towards that goal if you'd like :)

—
Reply to this email directly or view it on GitHub
#546 (comment).

[vmarmol]

Yes, you need to do both (I think we export the aliases today, if not we should). Here the aliases they're referring to are not a Docker container name or ID, is that correct @samek?

[rjnagal]

I see, the request is for exposing an env variable. This seems better handle in heapster aggregation where we can add arbitrary tags, or by exposing custom metrics hooks in cAdvisor.

@samek We do want to custom hooks to add extra columns, but we'll probably not get to it in near future.

[samek]

@vmarmol yes I'm not 100% sure but aliases are exported.
Problem is that there's no link to the marathon task id/app which is different.
And If you want to autoscale depending on the app which is run by marathon you would also need at least MARATHON_APP_ID to go with it.

Are you guys at all parsing json from the /var/lib/docker/containers/XXXXX/config.json ?

[vmarmol]

We don't read that file, but we read the libcontainer config here. This config does have the env vars so it should have the data you need.

[martensson]

Going to join the thread, having the same issue when trying to use cadvisor in combination with Marathon and Mesos.

[BrickXu]

+1

[mhausenblas]

@samek can you pls share your Marathon app spec (JSON) file for cAdvisor?

[samek]

@mhausenblas We're not using cadvisor anymore :(

For mesos monitoring and task monitoring from marathon we use
https://github.com/bobrik/collectd-docker
https://github.com/bobrik/docker-collectd-mesos

It solved all our problems.

If you need those I can post them for sure.

[mhausenblas]

Awesome, thanks @samek — yes, the Marathon app spec would be appreciated!

[scalp42]

@samek any chance?

[samek]

@scalp42 I've sent it directly to @mhausenblas since it's not related to cadvisor at all.
But sure, I'll just resend the mail to you.

[salimane]

@samek the marathon app spec could be put in a gist on github somewhere and it would be appreciated :)