timerfd: Too many open files, errno=24

[rolandjitsu]

Bug Report

Describe the bug

I'm seeing a lot of errors from Fluent Bit about too many open files (over a very long period).

I'm thinking this might be because it cannot reach the Fluentd server from time to time (logs are sent over a 4G connection, so the connection is not very reliable).

Right now ulimit -n is at 1024. Does this have to be higher?

To Reproduce

• Rubular link if applicable:
• Example log message if applicable:

May 24 21:36:06 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:36:06 kenpi fluent-bit[16999]: [2019/05/24 21:36:06] [ warn] net_tcp_fd_connect: getaddrinfo(host='myhost.com'): System error
May 24 21:36:06 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:36:06 kenpi fluent-bit[16999]: [2019/05/24 21:36:06] [error] [out_fw] no upstream connections available
May 24 21:37:48 kenpi fluent-bit[16999]: [2019/05/24 21:37:48] [  Error] timerfd: Too many open files, errno=24 at /tmp/fluent-bit/fluent-bit-1.1.1/lib/monkey/mk_core/mk_event_epoll.c:184
May 24 21:37:48 kenpi fluent-bit[16999]: [2019/05/24 21:37:48] [  Error] timerfd: Too many open files, errno=24 at /tmp/fluent-bit/fluent-bit-1.1.1/lib/monkey/mk_core/mk_event_epoll.c:184
May 24 21:37:48 kenpi fluent-bit[16999]: [2019/05/24 21:37:48] [  Error] timerfd: Too many open files, errno=24 at /tmp/fluent-bit/fluent-bit-1.1.1/lib/monkey/mk_core/mk_event_epoll.c:184
May 24 21:37:48 kenpi fluent-bit[16999]: [2019/05/24 21:37:48] [  Error] timerfd: Too many open files, errno=24 at /tmp/fluent-bit/fluent-bit-1.1.1/lib/monkey/mk_core/mk_event_epoll.c:184
May 24 21:37:48 kenpi fluent-bit[16999]: [2019/05/24 21:37:48] [  Error] timerfd: Too many open files, errno=24 at /tmp/fluent-bit/fluent-bit-1.1.1/lib/monkey/mk_core/mk_event_epoll.c:184
May 24 21:37:48 kenpi fluent-bit[16999]: [2019/05/24 21:37:48] [  Error] timerfd: Too many open files, errno=24 at /tmp/fluent-bit/fluent-bit-1.1.1/lib/monkey/mk_core/mk_event_epoll.c:184
May 24 21:37:48 kenpi fluent-bit[16999]: [2019/05/24 21:37:48] [  Error] timerfd: Too many open files, errno=24 at /tmp/fluent-bit/fluent-bit-1.1.1/lib/monkey/mk_core/mk_event_epoll.c:184
May 24 21:37:48 kenpi fluent-bit[16999]: [2019/05/24 21:37:48] [  Error] timerfd: Too many open files, errno=24 at /tmp/fluent-bit/fluent-bit-1.1.1/lib/monkey/mk_core/mk_event_epoll.c:184
May 24 21:37:48 kenpi fluent-bit[16999]: [2019/05/24 21:37:48] [  Error] timerfd: Too many open files, errno=24 at /tmp/fluent-bit/fluent-bit-1.1.1/lib/monkey/mk_core/mk_event_epoll.c:184
May 24 21:37:48 kenpi fluent-bit[16999]: socket: Too many open files
May 24 21:37:48 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:37:48 kenpi fluent-bit[16999]: [2019/05/24 21:37:48] [error] [io] could not create socket
May 24 21:37:48 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:37:48 kenpi fluent-bit[16999]: [2019/05/24 21:37:48] [error] [out_fw] no upstream connections available
May 24 21:37:48 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:37:48 kenpi fluent-bit[16999]: [2019/05/24 21:37:48] [ warn] net_tcp_fd_connect: getaddrinfo(host='myhost.com'): System error
May 24 21:37:48 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:37:48 kenpi fluent-bit[16999]: [2019/05/24 21:37:48] [error] [out_fw] no upstream connections available
May 24 21:38:16 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:38:16 kenpi fluent-bit[16999]: [2019/05/24 21:38:16] [ warn] net_tcp_fd_connect: getaddrinfo(host='myhost.com'): System error
May 24 21:38:58 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:38:58 kenpi fluent-bit[16999]: [2019/05/24 21:38:58] [ warn] net_tcp_fd_connect: getaddrinfo(host='myhost.com'): System error
May 24 21:38:58 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:38:58 kenpi fluent-bit[16999]: [2019/05/24 21:38:58] [error] [out_fw] no upstream connections available
May 24 21:38:58 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:38:58 kenpi fluent-bit[16999]: [2019/05/24 21:38:58] [ warn] net_tcp_fd_connect: getaddrinfo(host='myhost.com'): System error
May 24 21:38:58 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:38:58 kenpi fluent-bit[16999]: [2019/05/24 21:38:58] [error] [out_fw] no upstream connections available
May 24 21:38:58 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:38:58 kenpi fluent-bit[16999]: [2019/05/24 21:38:58] [ warn] net_tcp_fd_connect: getaddrinfo(host='myhost.com'): System error
May 24 21:38:58 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:38:58 kenpi fluent-bit[16999]: [2019/05/24 21:38:58] [error] [out_fw] no upstream connections available
May 24 21:39:04 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:39:04 kenpi fluent-bit[16999]: [2019/05/24 21:39:04] [ warn] net_tcp_fd_connect: getaddrinfo(host='myhost.com'): System error
May 24 21:46:40 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:46:40 kenpi fluent-bit[16999]: [2019/05/24 21:46:40] [ warn] net_tcp_fd_connect: getaddrinfo(host='myhost.com'): System error
May 24 21:46:50 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:46:50 kenpi fluent-bit[16999]: [2019/05/24 21:46:50] [ warn] net_tcp_fd_connect: getaddrinfo(host='myhost.com'): System error
May 24 21:47:00 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:47:00 kenpi fluent-bit[16999]: [2019/05/24 21:47:00] [ warn] net_tcp_fd_connect: getaddrinfo(host='myhost.com'): System error
May 24 21:47:11 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:47:11 kenpi fluent-bit[16999]: [2019/05/24 21:47:11] [ warn] net_tcp_fd_connect: getaddrinfo(host='myhost.com'): System error
May 24 21:47:11 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:47:11 kenpi fluent-bit[16999]: [2019/05/24 21:47:11] [error] [out_fw] no upstream connections available
May 24 21:47:11 kenpi fluent-bit[16999]: socket: Too many open files
May 24 21:47:11 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:47:11 kenpi fluent-bit[16999]: [2019/05/24 21:47:11] [error] [io] could not create socket
May 24 21:47:11 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:47:11 kenpi fluent-bit[16999]: [2019/05/24 21:47:11] [error] [out_fw] no upstream connections available
May 24 21:47:11 kenpi fluent-bit[16999]: socket: Too many open files
May 24 21:47:11 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:47:11 kenpi fluent-bit[16999]: [2019/05/24 21:47:11] [error] [io] could not create socket
May 24 21:47:11 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:47:11 kenpi fluent-bit[16999]: [2019/05/24 21:47:11] [error] [out_fw] no upstream connections available
May 24 21:47:11 kenpi fluent-bit[16999]: socket: Too many open files
May 24 21:47:11 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:47:11 kenpi fluent-bit[16999]: [2019/05/24 21:47:11] [error] [io] could not create socket
May 24 21:47:11 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:47:11 kenpi fluent-bit[16999]: [2019/05/24 21:47:11] [error] [out_fw] no upstream connections available
May 24 21:47:21 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:47:21 kenpi fluent-bit[16999]: [2019/05/24 21:47:21] [ warn] net_tcp_fd_connect: getaddrinfo(host='myhost.com'): System error
May 24 21:47:31 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:47:31 kenpi fluent-bit[16999]: [2019/05/24 21:47:31] [ warn] net_tcp_fd_connect: getaddrinfo(host='myhost.com'): System error
May 24 21:47:41 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:47:41 kenpi fluent-bit[16999]: [2019/05/24 21:47:41] [ warn] net_tcp_fd_connect: getaddrinfo(host='myhost.com'): System error
May 24 21:49:06 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:49:06 kenpi fluent-bit[16999]: [2019/05/24 21:49:06] [ warn] net_tcp_fd_connect: getaddrinfo(host='myhost.com'): System error
May 24 21:49:06 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:49:06 kenpi fluent-bit[16999]: [2019/05/24 21:49:06] [error] [out_fw] no upstream connections available
May 24 21:49:37 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:49:37 kenpi fluent-bit[16999]: [2019/05/24 21:49:37] [ warn] net_tcp_fd_connect: getaddrinfo(host='myhost.com'): System error
May 24 21:49:47 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:49:47 kenpi fluent-bit[16999]: [2019/05/24 21:49:47] [ warn] net_tcp_fd_connect: getaddrinfo(host='myhost.com'): System error
May 24 21:49:57 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:49:57 kenpi fluent-bit[16999]: [2019/05/24 21:49:57] [ warn] net_tcp_fd_connect: getaddrinfo(host='myhost.com'): System error
May 24 21:50:07 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:50:07 kenpi fluent-bit[16999]: [2019/05/24 21:50:07] [ warn] net_tcp_fd_connect: getaddrinfo(host='myhost.com'): System error
May 24 21:50:07 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:50:07 kenpi fluent-bit[16999]: [2019/05/24 21:50:07] [error] [out_fw] no upstream connections available
May 24 21:50:07 kenpi fluent-bit[16999]: socket: Too many open files
May 24 21:50:07 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:50:07 kenpi fluent-bit[16999]: [2019/05/24 21:50:07] [error] [io] could not create socket
May 24 21:50:07 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:50:07 kenpi fluent-bit[16999]: [2019/05/24 21:50:07] [error] [out_fw] no upstream connections available
May 24 21:50:07 kenpi fluent-bit[16999]: socket: Too many open files
May 24 21:50:07 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:50:07 kenpi fluent-bit[16999]: [2019/05/24 21:50:07] [error] [io] could not create socket
May 24 21:50:07 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:50:07 kenpi fluent-bit[16999]: [2019/05/24 21:50:07] [error] [out_fw] no upstream connections available
May 24 21:50:07 kenpi fluent-bit[16999]: socket: Too many open files
May 24 21:50:07 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:50:07 kenpi fluent-bit[16999]: [2019/05/24 21:50:07] [error] [io] could not create socket
May 24 21:50:07 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:50:07 kenpi fluent-bit[16999]: [2019/05/24 21:50:07] [error] [out_fw] no upstream connections available
May 24 21:50:07 kenpi fluent-bit[16999]: socket: Too many open files
May 24 21:50:07 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:50:07 kenpi fluent-bit[16999]: [2019/05/24 21:50:07] [error] [io] could not create socket
May 24 21:50:07 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:50:07 kenpi fluent-bit[16999]: [2019/05/24 21:50:07] [error] [out_fw] no upstream connections available
May 24 21:50:18 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:50:18 kenpi fluent-bit[16999]: [2019/05/24 21:50:18] [ warn] net_tcp_fd_connect: getaddrinfo(host='myhost.com'): System error
May 24 21:51:13 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:51:13 kenpi fluent-bit[16999]: [2019/05/24 21:51:13] [ warn] net_tcp_fd_connect: getaddrinfo(host='myhost.com'): System error
May 24 21:51:13 kenpi fluent-bit[16999]: [log] error opening log file /home/pi/logs/fluent-bit.log. Using stderr.
May 24 21:51:13 kenpi fluent-bit[16999]: [2019/05/24 21:51:13] [error] [out_fw] no upstream connections available
May 24 21:52:13 kenpi fluent-bit[16999]: socket: Too many open files

• Steps to reproduce the problem:

Expected behavior

I should not be seeing so many errors.

Screenshots

Your Environment

• Version used: v1.1.1
• Configuration: same as in High CPU usage on Raspberry Pi 3 #1341
• Environment name and version (e.g. Kubernetes? What version?):
• Server type and version:
• Operating System and version:

Linux raspi 4.14.44-v7+ #1117 SMP Thu May 31 16:57:56 BST 2018 armv7l GNU/Linux

PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
NAME="Raspbian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

• Filters and plugins:

Additional context

[gwkunze]

I'm running into this as well, after having a temporary failure on the destination fluentd servers a lot of files were created in /var/spool/td-agent-bit/ and for some reason fluent-bit keeps an open fd on every one of these. I hope that is a bug and not necessary as that seems pretty inefficient.

[edsiper]

If you are using file system based storage this is already fixed for major release v1.2 to be available around next week.

[rolandjitsu]

@edsiper I am using the tail plugin + the sqlite db enabled to track last sent log.

[rolandjitsu]

I'm seeing these errors again. It usually happens when FB cannot reach the upstream fluentd server.

Running cat /proc/sys/fs/file-max gives 75611. That should probably be enough.

I'm already on v1.2.2.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.