You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We are using Banzaicloud Logging Operator on Openshift to provide log forwarding in a multi tenant way.
Fluentd sends all logs to a Kafka with credentials per user.
On our testbed platform, many users still have a fluentd configuration but don't have valid credentials resulting in a buffer directory (/buffers/ in our case) that is growing more and more despite the fluentd buffer configuration to drop logs after a certain amount of time.
As 1 log line creates 2 files in /buffers, we have hundred of thousand logs/files stuck in that /buffer directory and a simple ls /buffers can take 2-5minutes.
The more file there is in the buffer directory, the less fluentd is responding to new connections (fluentbit or prometheus scraping).
I beg there could be a kind of freeze of the ruby process while it's waiting the OS to execute some command in /buffers ?
To Reproduce
Create an output to fluentd that is rejecting logs and start accumulate logs in fluentd buffer directory.
Try to connect to fluentd listening sockets.
Expected behavior
Fluentd not rejecting connections even if buffer is heavy loaded
This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 7 days
Describe the bug
We are using Banzaicloud Logging Operator on Openshift to provide log forwarding in a multi tenant way.
Fluentd sends all logs to a Kafka with credentials per user.
On our testbed platform, many users still have a fluentd configuration but don't have valid credentials resulting in a buffer directory (/buffers/ in our case) that is growing more and more despite the fluentd buffer configuration to drop logs after a certain amount of time.
As 1 log line creates 2 files in /buffers, we have hundred of thousand logs/files stuck in that /buffer directory and a simple
ls /buffers
can take 2-5minutes.The more file there is in the buffer directory, the less fluentd is responding to new connections (fluentbit or prometheus scraping).
I beg there could be a kind of freeze of the ruby process while it's waiting the OS to execute some command in /buffers ?
To Reproduce
Create an output to fluentd that is rejecting logs and start accumulate logs in fluentd buffer directory.
Try to connect to fluentd listening sockets.
Expected behavior
Fluentd not rejecting connections even if buffer is heavy loaded
Your Environment
Your Configuration
Your Error Log
None, error logs are in fluentd clients like fluentbit: log-collector-ocp-1-fluentbit-8xqh7 fluent-bit [2022/11/29 14:51:52] [error] [net] TCP connection failed: log-collector-ocp-1-fluentd.log-collector.svc:24240 (Connection refused)
Additional context
No response
The text was updated successfully, but these errors were encountered: