Open
Description
openedon Sep 22, 2024
Before creating an issue, make sure you've checked the following:
- You are running the latest released version of k0s
- Make sure you've searched for existing issues, both open and closed
- Make sure you've searched for PRs too, a fix might've been merged already
- You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.
Platform
as360@AS360-AIO-Ubuntu:~$ uname -srvmo; cat /etc/os-release || lsb_release -a
Linux 6.8.0-45-generic #45-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 30 12:02:04 UTC 2024 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logoVersion
v1.30.4+k0s.0
Sysinfo
`k0s sysinfo`
Total memory: 35.2 GiB (pass)
Disk space available for /var/lib/k0s: 197.0 GiB (pass)
Name resolution: localhost: [::1 127.0.0.1] (pass)
Operating system: Linux (pass)
Linux kernel release: 6.8.0-45-generic (pass)
Max. file descriptors per process: current: 1048576 / max: 1048576 (pass)
AppArmor: unavailable (pass)
Executable in PATH: modprobe: /sbin/modprobe (pass)
Executable in PATH: mount: /bin/mount (pass)
Executable in PATH: umount: /bin/umount (pass)
/proc file system: mounted (0x9fa0) (pass)
Control Groups: version 2 (pass)
cgroup controller "cpu": available (is a listed root controller) (pass)
cgroup controller "cpuacct": available (via cpu in version 2) (pass)
cgroup controller "cpuset": available (is a listed root controller) (pass)
cgroup controller "memory": available (is a listed root controller) (pass)
cgroup controller "devices": available (device filters attachable) (pass)
cgroup controller "freezer": available (cgroup.freeze exists) (pass)
cgroup controller "pids": available (is a listed root controller) (pass)
cgroup controller "hugetlb": available (is a listed root controller) (pass)
cgroup controller "blkio": available (via io in version 2) (pass)
CONFIG_CGROUPS: Control Group support: no kernel config found (warning)
CONFIG_NAMESPACES: Namespaces support: no kernel config found (warning)
CONFIG_NET: Networking support: no kernel config found (warning)
CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: no kernel config found (warning)
CONFIG_PROC_FS: /proc file system support: no kernel config found (warning)
What happened?
K0S running on a single node system (multiple services run by docker-compose) doesn't survive multiple reboots (it comes up after a few, and then it doesn't, or it doesn't come up at all after some reboots).
Attached below a sample docker-compose file to demo the problem.
Tried on Ubuntu 24.04 and CentOS 9 - same results
Steps to reproduce
- Take the sample docker-compose file (attached below)
- Run the following command:
docker compose -f aio-compose-sample.yaml up -d --wait - Reboot the host several times, at some point, after a handful of restarts (or even after the first restart), the K0S gets broken
Expected behavior
The K0S should survive restarts, always.
Actual behavior
After a few restarts, the K0S breaks down:
# docker compose -f aio-compose-sample.yaml exec k0s k0s kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
cert-manager cert-manager-9647b459d-hlxr2 1/1 Running 1 (4h3m ago) 4h15m
cert-manager cert-manager-cainjector-5d8798687c-h8lk4 1/1 Running 2 (4h3m ago) 4h15m
cert-manager cert-manager-webhook-c77744d75-b5vcn 1/1 Running 1 (4h3m ago) 4h15m
ingress-nginx ingress-nginx-admission-create-bxxgb 0/1 Pending 0 17m
ingress-nginx ingress-nginx-admission-create-lh4p7 0/1 Terminating 0 3h58m
ingress-nginx ingress-nginx-controller-55df698df5-6vtxj 1/1 Running 1 (4h3m ago) 4h16m
k0s-system k0s-pushgateway-86bd768578-cp7cq 1/1 Running 1 (4h3m ago) 4h17m
kube-system coredns-85c69f454c-2hgn7 1/1 Running 1 (4h3m ago) 4h17m
kube-system konnectivity-agent-27m8k 1/1 Terminating 1 (4h3m ago) 4h17m
kube-system kube-proxy-pxnl5 1/1 Running 1 (4h3m ago) 4h17m
kube-system kube-router-84vsr 1/1 Terminating 1 (4h3m ago) 4h17m
kube-system metrics-server-7cc78958fc-gkj7l 1/1 Running 1 (4h3m ago) 4h17m
openebs openebs-localpv-provisioner-86d8949887-49rr7 1/1 Running 0 4h2m
openebs openebs-pre-upgrade-hook-6jcts 0/1 Pending 0 3h38m
# docker compose -f aio-compose-sample.yaml exec k0s k0s kubectl get nodes
NAME STATUS ROLES AGE VERSION
k0s NotReady control-plane 4h18m v1.30.4+k0s
Screenshots and logs
Kindly advise what logs are needed, and I'll be happy to add them.
Additional context
Adding a sample docker compose to demonstrate the problem:
aio-compose-sample.zip
Docker version info:
as360@AS360-AIO-Ubuntu:~$ docker compose -f aio-compose-sample.yaml exec k0s k0s status
Version: v1.30.4+k0s.0
Process ID: 7
Role: controller
Workloads: true
SingleNode: false
Kube-api probing successful: true
Kube-api probing last error:
as360@AS360-AIO-Ubuntu:~$ docker version
Client: Docker Engine - Community
Version: 27.3.1
API version: 1.47
Go version: go1.22.7
Git commit: ce12230
Built: Fri Sep 20 11:40:59 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 27.3.1
API version: 1.47 (minimum version 1.24)
Go version: go1.22.7
Git commit: 41ca978
Built: Fri Sep 20 11:40:59 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.22
GitCommit: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
runc:
Version: 1.1.14
GitCommit: v1.1.14-0-g2c9f560
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
