Fluentd stops processing the logs if a pod is restarted

[dhineshbabuelango]

Hi Team,
We are using fluentd in EKS cluster, we are injesting logs from a file in the container to fluentd through a configmap, using grok parser, while the initial setup working fine everytime when a deployment or pod is restarted, we had to recreate the config map(without any change) as well only then fluentd processing the logs

[Cryptophobia]

This is a known issue with file-mount logs.

See #289

Duplicate of #289

[Cryptophobia]

Duplicate of #289

[Cryptophobia]

docker.io/vmware/kube-fluentd-operator:v1.16.2 may fix your problems if you don't run very large clusters with lots of configs.

[dhineshbabuelango]

Is this issue only with type mounted-file?, if so can I use any other type.
Note: the logs of my container are piped to a different folder like /app/jboss/logs and not the stdout and stderr. So in this case my only option is mounted-file?

[Cryptophobia]

Is this issue only with type mounted-file?, if so can I use any other type.

Yes, it only affects mounted-file types.

We are working on a resolution that may work. Are you able to switch to logging to stdout for the jboss server in the container/pod? If you do that then you will not face this problem when pods restart.

[dhineshbabuelango]

@Cryptophobia Thanks for the update, we are in process of moving the logs to stdout but since we have some applications already running in prod we are finding for a fix. The mounted-file plugin only supports emptydir volume or we can use a persistant storage.

[dhineshbabuelango]

@Cryptophobia If I restart teh configmap in few seconds to minutes, will fluentd be able to capture all the logs without any miss?

[Cryptophobia]

@Cryptophobia Thanks for the update, we are in process of moving the logs to stdout but since we have some applications already running in prod we are finding for a fix. The mounted-file plugin only supports emptydir volume or we can use a persistant storage.

@dhineshbabuelango , KFO only supports mounted-files logs as emptyDir types so persistent storage would not work at this time. When KFO makes representation in its memory for each of the pods, it only takes into account volumetypes that are emptyDir types.

@Cryptophobia If I restart teh configmap in few seconds to minutes, will fluentd be able to capture all the logs without any miss?

If you write a hook to reload the configmap to delete and recreate the configmap right away after restarting the pods, it should work. It needs to happen right after the restart of the pods though.

[dhineshbabuelango]

@Cryptophobia thanks for your response. Will refreshing(delete & recreate) the configmap will lead to fluentd missing any of the logs that was currently reading during the restart ?