too many open files

[gricuk]

We're gettingtoo many open fileserror in few clusters with image ghcr.io/banzaicloud/fluentd:v1.14.5-alpine-1 and higher.

It happens randomly and I can't provide any conditions, but could you please check the following PR fluent/fluentd#3844. It was fixed in 1.15.2 release.

Now we have downgraded fluentd to ghcr.io/banzaicloud/fluentd:v1.14.4-alpine-2 and all is okay with this version (but I'm not sure because as I mentioned above it happens randomly and not all clusters have this error)

Environment details:

• Kubernetes version v1.21.12+rke2r2:
• Cloud-provider/provisioner (e.g. AKS, GKE, EKS, PKE etc): RKE2
• logging-operator version (e.g. 2.1.1): logging-operator-3.17.9
• Install method (e.g. helm or static manifests): helm

/kind bug

[1it]

It could be caused by chunk_limit_size: 8MB, which is set here. This actually a nonsense, it should be at least 256MB (Fluentd default). 8MB is a default value for in memory buffer, not for file based.
I've had about 32k buffer files in one instance because of that and constant too many open files errors.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!

[pepov]

Thanks for reporting this and sorry for the late reply. If I remember correctly previously fluentd used 256M for memory buffers by default (or at least it was the case in some old versions), which caused OOM issues, that was the reason why we did this:
#751

I beleive we can remove this explicit default and rely on the fluentd settings if we can verify our supported fluentd images now does it correctly.

Scheduling this to the next minor release.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!

[pepov]

According to the docs, the chunk_limit_size is now smarter as it uses 8MB for memory and 256MB for file buffers by default:
https://docs.fluentd.org/configuration/buffer-section#buffering-parameters

We can remove our default setting to rely on fluentd's settings.