Description
On CentOS linux distributions using SELinux the systemd unit doesn't have the required permissions to set the maxclients config via setrlimit(): https://github.com/Snapchat/KeyDB/blob/main/src/server.cpp#L3546
Below are the logs when the service was started:
2342:2342:M 25 Oct 2022 15:20:34.959 # Server can't set maximum open files to 65000 because of OS error: Operation not permitted.
2342:2342:M 25 Oct 2022 15:20:34.959 # Current maximum open files is 10240. maxclients has been reduced to 10208 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
This happened even though we tuned the kernel with the below:
[root@ip-10-0-1-248 ~]# ulimit -n
500000
After inspecting the systemd unit configured on /lib/systemd/system/keydb.service and reading more about the CAP_SYS_RESOURCE setting on CapabilityBoundingSet this "should have worked". However, after arm-wrestling with this issue for a little while it only worked after I added:
AmbientCapabilities=CAP_SETGID CAP_SETUID CAP_SYS_RESOURCE
https://www.freedesktop.org/software/systemd/man/systemd.exec.html#AmbientCapabilities=
Here's my entire systemd unit configuration:
[Unit]
Description=Advanced key-value store
After=network.target
Documentation=https://docs.keydb.dev, man:keydb-server(1)
[Service]
Type=notify
ExecStart=/usr/bin/keydb-server /etc/keydb/keydb.conf
ExecStop=/bin/kill -s TERM $MAINPID
PIDFile=/var/run/keydb/keydb-server.pid
TimeoutStopSec=0
Restart=always
User=keydb
Group=keydb
RuntimeDirectory=keydb
RuntimeDirectoryMode=2755
UMask=007
PrivateTmp=yes
LimitNOFILE=65535
PrivateDevices=yes
ProtectHome=yes
ReadOnlyDirectories=/
ReadWriteDirectories=-/var/lib/keydb
ReadWriteDirectories=-/var/log/keydb
ReadWriteDirectories=-/var/run/keydb
NoNewPrivileges=true
CapabilityBoundingSet=CAP_SETGID CAP_SETUID CAP_SYS_RESOURCE
AmbientCapabilities=CAP_SETGID CAP_SETUID CAP_SYS_RESOURCE
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
# keydb-server can write to its own config file when in cluster mode so we
# permit writing there by default. If you are not using this feature, it is
# recommended that you replace the following lines with "ProtectSystem=full".
ProtectSystem=true
ReadWriteDirectories=-/etc/keydb
[Install]
WantedBy=multi-user.target
Alias=keydb.service
Please note: I also had to adjust the Type to notify as I currently have Monit installed to verify the service health. It looks like this was worked by your team at some point but it was not made the default value: #352
KeyDB log after successful service restart:
1380:signal-handler (1666729774) Received SIGTERM scheduling shutdown...
1380:1388:M 25 Oct 2022 20:29:34.770 # User requested shutdown...
1380:1388:M 25 Oct 2022 20:29:34.770 * Removing the pid file.
1380:1388:M 25 Oct 2022 20:29:34.770 # KeyDB is now ready to exit, bye bye...
1503:1503:C 25 Oct 2022 20:29:34.906 # WARNING supervised by systemd - you MUST set appropriate values for TimeoutStartSec and TimeoutStopSec in your service unit.
1503:1503:C 25 Oct 2022 20:29:34.906 # oO0OoO0OoO0Oo KeyDB is starting oO0OoO0OoO0Oo
1503:1503:C 25 Oct 2022 20:29:34.906 # KeyDB version=6.3.1, bits=64, commit=ee16abf0, modified=0, pid=1503, just started
1503:1503:C 25 Oct 2022 20:29:34.906 # Configuration loaded
1503:1503:M 25 Oct 2022 20:29:34.908 * Increased maximum number of open files to 65032 (it was originally set to 10240).