Open
Description
openedon Oct 15, 2024
Before creating an issue, make sure you've checked the following:
- You are running the latest released version of k0s
- Make sure you've searched for existing issues, both open and closed
- Make sure you've searched for PRs too, a fix might've been merged already
- You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.
Platform
No response
Version
v1.29.9+k0s
Sysinfo
`k0s sysinfo`
➡️ Please replace this text with the output of `k0s sysinfo`. ⬅️
What happened?
When a third controller is added the leader lease is somehow lost and when it is re-acquired the applier-manager is not restarted resulting in updates to manifests or stacks not being applied.
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: I1015 16:55:27.386215 3978 leaderelection.go:285] failed to renew lease kube-node-lease/k0s-endpoint-reconciler: timed out waiting for the condition
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: I1015 16:55:27.386295 3978 leaderelection.go:285] failed to renew lease kube-node-lease/k0s-ctrl-node-e3a0d-00: timed out waiting for the condition
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="Lost leader lease" component=controllerlease
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: I1015 16:55:27.391034 3978 leaderelection.go:250] attempting to acquire leader lease kube-node-lease/k0s-ctrl-node-e3a0d-00...
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="Lost leader lease" component=poolleaderelector
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: I1015 16:55:27.391062 3978 leaderelection.go:250] attempting to acquire leader lease kube-node-lease/k0s-endpoint-reconciler...
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="lost leader lease" component=poolleaderelector
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=error msg="lost leader lease, this should not really happen!?!?!?" component=controllerlease
...
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="manifest watcher done" component=applier-manager
...
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="Acquired leader lease" component=poolleaderelector
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="acquired leader lease" component=poolleaderelector
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="Acquired leader lease" component=controllerlease
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="acquired leader lease" component=controllerlease
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="Acquired leader lease" component=extensions_controller
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="Acquired leader lease" component=extensions_controller
...
Steps to reproduce
Expected behavior
Changes to manifests dir will continue to be applied to the cluster
Actual behavior
Changes are no longer reflected in the cluster.
Screenshots and logs
k0scontroller-logs.txt
k0scontroller-logs.txt
k0scontroller-logs.txt
Additional context
No response
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment