Promtail remove file watch code and only rely on periodic sync?

[slim-bean]

After adding the periodic sync function in #293, the fsnotify.Watcher code is more or less redundant and adds complexity, I think we could probably remove it.

This would result in a delay before new files would be sent based on what they polling rate of the new sync function is, e.g. up to 10 seconds in the worst case if the sync function polling interval was set to 10 seconds, though nominally the delay would be around 1/2 the sync delay.

[kfox1111]

some containers are very short lived, for example initContainers. It is nice to get logs from them quickly. Also, when log rolling is enabled in the cluster, its good to detect the new files quickly or else whole files of logs could be lost when a quick burst of logging happens. or for k8s jobs like, build jobs, that are short lived. the more logs you can get out of them the better.

So, I think its beneficial to keep the watches.

[woodsaj]

I think we should keep the watch. 10seconds is a long time to go without noticing there is a new log to tail.

when containers crash and restart, kubelet creates a new log file, eg if the current log is 0.log when the container is restarted 1.log is created. we should start tailing that straight away.

I think the fsnotify events should be the primary trigger for new files to tail. The periodic sync is the backup.

[tomwilkie]

I suggested to @slim-bean we remove the code as it was getting a bit complicated, but Ed simplified the PR so we don't need to remove it anymore.