KernelMonitor: Add journald support

[Random-Liu]

Add journald support.

Consider reusing the code in cadvisor, and maybe replace hpcloud/tail with the simpler implementation in cadvisor #3

[adohe-zz]

@Random-Liu should we consider priority between journaled and kernel log file?

[Random-Liu]

@adohe Sorry, I didn't get your question. We only parse kernel log now, even using journald we also only care about kernel log inside. :)

[adohe-zz]

@Random-Liu sorry about not explain very clearly, I mean whether we should consider the priority, just like cadvisor, first will try systemd, and then some log files.

[Random-Liu]

@adohe Yeah, I think so. If there is journald, we should use it first. :)

[derekwaynecarr]

@dchen1107 @Random-Liu @adohe -- is adding journald support on either of your roadmap for Kubernetes 1.5? For example, would this be needed if/when you look to upgrade GCE node images?

/cc @hodovska

[Random-Liu]

@derekwaynecarr @dchen1107 I think we've to support journald if we are switching to GCI.

[adohe-zz]

@Random-Liu I think I have did some research and want to give a retry. Will open a PR asap.

[Random-Liu]

@adohe Cool! It's very appreciated!
Just curious, how are you going to do that? Use go-systemd or mount journalctl binary or something? :)

[adohe-zz]

@Random-Liu I was using go-systemd, I have researched its code.

[Random-Liu]

@adohe That's great! This will be quite useful!

Not only in node problem detector, we also have this problem in cadvisor (https://github.com/google/cadvisor/blob/381f24b4f22badecade6a51ed7539f11a926175e/utils/oomparser/oomparser.go#L168-L178). It is using journalctl binary directly, which IIUC doesn't actually work inside container google/cadvisor#645 given the cadvisor docker run command:

sudo docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:rw \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --publish=8080:8080 \
  --detach=true \
  --name=cadvisor \
  google/cadvisor:latest

If you are interested, I think you can also update cadvisor. :)

[adohe-zz]

Thanks @Random-Liu I will give a retry after finish this one. :)

[zihaoyu]

@adohe @Random-Liu Any update on this one? I'm running Kubernetes on CoreOS, which uses journald. I'm running a Docker image I built from master, but it looks like it is not watching journald logs.

[Random-Liu]

@adohe @Random-Liu Any update on this one? I'm running Kubernetes on CoreOS, which uses journald. I'm running a Docker image I built from master, but it looks like it is not watching journald logs.

This is not completely done yet. I'm working on a PR recently, and may send the PR this week.

[zihaoyu]

@Random-Liu Thanks. Let me know if/how I can help. I'm going to test it in our staging cluster once it's ready.