Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NO_FILE ulimit still seems to be different to v1.4.1 #7206

Closed
scrayos opened this issue May 11, 2023 · 1 comment · Fixed by #7208
Closed

NO_FILE ulimit still seems to be different to v1.4.1 #7206

scrayos opened this issue May 11, 2023 · 1 comment · Fixed by #7208

Comments

@scrayos
Copy link

scrayos commented May 11, 2023

Bug Report

Description

The bug that was described here seems to be still present to some degree. At least the NO_FILE behavior seems to be still different to the behavior in Talos Linux v1.4.1.

I've deployed sonarqube in my cluster and (aside from the database) this setup only involves a single Pod (that is part of a Statefulset). The pod uses a busybox container to set the required sysctls and ulimits for sonarqube/elasticsearch.

But it seems like those changes will no longer be present in the other container (sonarqube). The script in the init-container looks like this:

if [[ "$(sysctl -n vm.max_map_count)" -lt 524288 ]]; then
  sysctl -w vm.max_map_count=524288
fi
if [[ "$(sysctl -n fs.file-max)" -lt 131072 ]]; then
  sysctl -w fs.file-max=131072
fi
if [[ "$(ulimit -n)" != "unlimited" ]]; then
  if [[ "$(ulimit -n)" -lt 131072 ]]; then
    echo "ulimit -n 131072"
    ulimit -n 131072
  fi
fi
if [[ "$(ulimit -u)" != "unlimited" ]]; then
  if [[ "$(ulimit -u)" -lt 8192 ]]; then
    echo "ulimit -u 8192"
    ulimit -u 8192
  fi
fi

but the output of ulimit -a in the sonarqube container is (notice open files = 1024):

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 256511
max locked memory       (kbytes, -l) 8192
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 256511
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

I've changed nothing but the Talos version and this pod is no longer working, failing with a message, that is saying, that max files is not high enough. So it seems that now changes to sysctl for NO_FILES in other containers won't affect the rest of the Pod anymore.

Environment

  • Talos version: 1.4.3
  • Kubernetes version:
    Client Version: v1.27.1
    Kustomize Version: v5.0.1
    Server Version: v1.27.1
  • Platform: Hetzner Cloud/Bare Metal
@frezbo
Copy link
Member

frezbo commented May 11, 2023

Yes, the bug is still not resolved, we're looking into it

frezbo added a commit to frezbo/talos that referenced this issue May 11, 2023
This reverts commit a2565f6.

The fix done in `a2565f67`, was actually a no-op caused by the
misunderstanding the fix done in Go and backported to [Go 1.20.4](golang/go@ecf7e00).
The fix gave a false confidence that it was working when it was tested
against Talos `main` branch since the PR siderolabs#7190 bumped `x/sys` package
from [v0.7.0 -> v0.8.0](golang/go@ecf7e00), the actual change in `x/sys` can be found here at https://cs.opensource.google/go/x/sys/+/ff18efa0a3fa22d4fede381b822bbcfe53b7ee7c which meant that when updating Go to 1.20.4 the `x/sys` package should been updated too. The `x/sys` package changed how the syscall to set the rlimit was called, it got moved into the Go stdlib instead of calling rlimit syscall in the `x/sys` package, which meant a combination of using Go 1.20.4 and an older `x/sys` package means `RLIMIT_NOFILE` value would not be set back to the original value.

The Talos 1.4 release branch currently have  `x/sys`
at [v0.7.0(https://github.com/siderolabs/talos/blob/v1.4.3/go.mod#L133),
so the backport would consist of this change along another commit bumping `x/sys` package to `v0.8.0`.

Fixes: siderolabs#7198
Fixes: siderolabs#7206

Co-authored-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>
Signed-off-by: Noel Georgi <git@frezbo.dev>
(cherry picked from commit 4f720d4)
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 13, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants