Closed
Description
openedon Aug 15, 2024
Before creating an issue, make sure you've checked the following:
- You are running the latest released version of k0s
- Make sure you've searched for existing issues, both open and closed
- Make sure you've searched for PRs too, a fix might've been merged already
- You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.
Platform
Linux 6.5.0-1025-gcp #27~22.04.1-Ubuntu SMP Tue Jul 16 23:03:39 UTC 2024 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammyVersion
v1.29.7+k0s.0
Sysinfo
`k0s sysinfo`
Machine ID: "29cea7f8af2ab6c444e8ecf6b8e9aec00374c3f2db06b285f4f0f7070078c04e" (from machine) (pass)
Total memory: 29.4 GiB (pass)
Disk space available for /var/lib/k0s: 168.8 GiB (pass)
Name resolution: localhost: [127.0.0.1] (pass)
Operating system: Linux (pass)
Linux kernel release: 6.5.0-1025-gcp (pass)
Max. file descriptors per process: current: 1048576 / max: 1048576 (pass)
AppArmor: active (pass)
Executable in PATH: modprobe: /usr/sbin/modprobe (pass)
Executable in PATH: mount: /usr/bin/mount (pass)
Executable in PATH: umount: /usr/bin/umount (pass)
/proc file system: mounted (0x9fa0) (pass)
Control Groups: version 2 (pass)
cgroup controller "cpu": available (is a listed root controller) (pass)
cgroup controller "cpuacct": available (via cpu in version 2) (pass)
cgroup controller "cpuset": available (is a listed root controller) (pass)
cgroup controller "memory": available (is a listed root controller) (pass)
cgroup controller "devices": available (device filters attachable) (pass)
cgroup controller "freezer": available (cgroup.freeze exists) (pass)
cgroup controller "pids": available (is a listed root controller) (pass)
cgroup controller "hugetlb": available (is a listed root controller) (pass)
cgroup controller "blkio": available (via io in version 2) (pass)
CONFIG_CGROUPS: Control Group support: built-in (pass)
CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass)
CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass)
CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass)
CONFIG_CPUSETS: Cpuset support: built-in (pass)
CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass)
CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass)
CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass)
CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
CONFIG_NAMESPACES: Namespaces support: built-in (pass)
CONFIG_UTS_NS: UTS namespace: built-in (pass)
CONFIG_IPC_NS: IPC namespace: built-in (pass)
CONFIG_PID_NS: PID namespace: built-in (pass)
CONFIG_NET_NS: Network namespace: built-in (pass)
CONFIG_NET: Networking support: built-in (pass)
CONFIG_INET: TCP/IP networking: built-in (pass)
CONFIG_IPV6: The IPv6 protocol: built-in (pass)
CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
CONFIG_NF_CONNTRACK: Netfilter connection tracking support: built-in (pass)
CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: module (pass)
CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass)
CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass)
CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass)
CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass)
CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
CONFIG_NETFILTER_NETLINK: built-in (pass)
CONFIG_NF_NAT: module (pass)
CONFIG_IP_SET: IP set support: module (pass)
CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
CONFIG_IP_VS: IP virtual server support: module (pass)
CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
CONFIG_IP_VS_SH: Source hashing scheduling: module (pass)
CONFIG_IP_VS_RR: Round-robin scheduling: module (pass)
CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass)
CONFIG_NF_CONNTRACK_IPV4: IPv4 connetion tracking support (required for NAT): unknown (warning)
CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass)
CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
CONFIG_NF_DEFRAG_IPV4: built-in (pass)
CONFIG_NF_CONNTRACK_IPV6: IPv6 connetion tracking support (required for NAT): unknown (warning)
CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass)
CONFIG_IP6_NF_FILTER: Packet filtering: module (pass)
CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass)
CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
CONFIG_NF_DEFRAG_IPV6: built-in (pass)
CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass)
CONFIG_LLC: module (pass)
CONFIG_STP: module (pass)
CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: built-in (pass)
CONFIG_PROC_FS: /proc file system support: built-in (pass)
What happened?
My chart upgrade failed. This initially shows up in the status
status:
appVersion: 2.8.1
error: "can't upgrade loadedChart `docker-registry`: an error occurred while rolling
back the release. original upgrade error: failed to replace object: PersistentVolumeClaim
\"registry\" is invalid: spec: Forbidden: spec is immutable after creation except
resources.requests and volumeAttributesClassName for bound claims\n core.PersistentVolumeClaimSpec{\n \tAccessModes:
\ {\"ReadWriteOnce\"},\n \tSelector: nil,\n \tResources: {Requests:
{s\"storage\": {i: {...}, s: \"10Gi\", Format: \"BinarySI\"}}},\n- \tVolumeName:
\ \"pvc-cadec2a9-d4f7-441d-bf44-0a2925bc8a2e\",\n+ \tVolumeName: \"\",\n \tStorageClassName:
&\"openebs-hostpath\",\n \tVolumeMode: &\"Filesystem\",\n \t... // 3 identical
fields\n }\n: failed to replace object: PersistentVolumeClaim \"registry\" is
invalid: spec: Forbidden: spec is immutable after creation except resources.requests
and volumeAttributesClassName for bound claims\n core.PersistentVolumeClaimSpec{\n \tAccessModes:
\ {\"ReadWriteOnce\"},\n \tSelector: nil,\n \tResources: {Requests:
{s\"storage\": {i: {...}, s: \"10Gi\", Format: \"BinarySI\"}}},\n- \tVolumeName:
\ \"pvc-cadec2a9-d4f7-441d-bf44-0a2925bc8a2e\",\n+ \tVolumeName: \"\",\n \tStorageClassName:
&\"openebs-hostpath\",\n \tVolumeMode: &\"Filesystem\",\n \t... // 3 identical
fields\n }\n"
After a while the error goes away in the chart status
status:
appVersion: 2.8.1
namespace: registry
releaseName: docker-registry
revision: 1
updated: 2024-08-15 12:22:52.820544519 +0000 UTC m=+167.490367477
valuesHash: 6a4960ff9c68791c1f38efc1793ec5f881d312422d26dd3cfd3320dad93f8d97
version: 2.2.3
If I look at the helm release it clearly didnt roll out
root@ethanm-ec-1:/home/ethan# helm -n registry history docker-registry
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/lib/k0s/pki/admin.conf
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Thu Aug 15 11:58:11 2024 deployed docker-registry-2.2.3 2.8.1 Install complete
2 Thu Aug 15 12:20:57 2024 superseded docker-registry-2.2.3 2.8.1 Upgrade "docker-registry" failed: failed to replace object: PersistentVolumeClaim "registry" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound clai...
3 Thu Aug 15 12:20:58 2024 failed docker-registry-2.2.3 2.8.1 Rollback "docker-registry" failed: failed to replace object: PersistentVolumeClaim "registry" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound cla...
What seems to happen is that eventually the reconciliation happens again because of the error, sees that the chart does not need to be updated and resets the error.
Aug 15 12:20:57 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:57" level=debug msg="Install or update reconciliation request: kube-system/k0s-addon-chart-docker-registry" component=extensions_controller extensions_type=helm
Aug 15 12:20:57 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:57" level=debug msg="preparing upgrade for docker-registry" component=helm
Aug 15 12:20:57 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:57" level=debug msg="performing update for docker-registry" component=helm
Aug 15 12:20:57 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:57" level=debug msg="creating upgraded release for docker-registry" component=helm
Aug 15 12:20:57 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:57" level=debug msg="checking 5 resources for changes" component=helm
Aug 15 12:20:57 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:57" level=debug msg="Replaced \"registry-secret\" with kind Secret for kind Secret" component=helm
Aug 15 12:20:57 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:57" level=debug msg="Replaced \"registry-config\" with kind ConfigMap for kind ConfigMap" component=helm
Aug 15 12:20:57 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:57" level=debug msg="error updating the resource \"registry\":\n\t failed to replace object: PersistentVolumeClaim \"registry\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims\n\u00a0\u00a0core.PersistentVolumeClaimSpec{\n\u00a0\u00a0\tAccessModes: {\"ReadWriteOnce\"},\n\u00a0\u00a0\tSelector: nil,\n\u00a0\u00a0\tResources: {Requests: {s\"storage\": {i: {...}, s: \"10Gi\", Format: \"BinarySI\"}}},\n-\u00a0\tVolumeName: \"pvc-cadec2a9-d4f7-441d-bf44-0a2925bc8a2e\",\n+\u00a0\tVolumeName: \"\",\n\u00a0\u00a0\tStorageClassName: &\"openebs-hostpath\",\n\u00a0\u00a0\tVolumeMode: &\"Filesystem\",\n\u00a0\u00a0\t... // 3 identical fields\n\u00a0\u00a0}\n" component=helm
Aug 15 12:20:57 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:57" level=debug msg="Replaced \"registry\" with kind Service for kind Service" component=helm
Aug 15 12:20:57 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:57" level=debug msg="Replaced \"registry\" with kind Deployment for kind Deployment" component=helm
Aug 15 12:20:57 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:57" level=info msg="I0815 12:20:57.947235 950619 event.go:376] \"Event occurred\" object=\"registry/registry\" fieldPath=\"\" kind=\"Deployment\" apiVersion=\"apps/v1\" type=\"Normal\" reason=\"ScalingReplicaSet\" message=\"Scaled up replica set registry-764b9d8cc4 to 1\"" component=kube-controller-manager stream=stderr
Aug 15 12:20:57 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:57" level=info msg="I0815 12:20:57.953452 950619 event.go:376] \"Event occurred\" object=\"registry/registry-764b9d8cc4\" fieldPath=\"\" kind=\"ReplicaSet\" apiVersion=\"apps/v1\" type=\"Normal\" reason=\"SuccessfulCreate\" message=\"Created pod: registry-764b9d8cc4-wm26m\"" component=kube-controller-manager stream=stderr
Aug 15 12:20:57 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:57" level=debug msg="warning: Upgrade \"docker-registry\" failed: failed to replace object: PersistentVolumeClaim \"registry\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims\n\u00a0\u00a0core.PersistentVolumeClaimSpec{\n\u00a0\u00a0\tAccessModes: {\"ReadWriteOnce\"},\n\u00a0\u00a0\tSelector: nil,\n\u00a0\u00a0\tResources: {Requests: {s\"storage\": {i: {...}, s: \"10Gi\", Format: \"BinarySI\"}}},\n-\u00a0\tVolumeName: \"pvc-cadec2a9-d4f7-441d-bf44-0a2925bc8a2e\",\n+\u00a0\tVolumeName: \"\",\n\u00a0\u00a0\tStorageClassName: &\"openebs-hostpath\",\n\u00a0\u00a0\tVolumeMode: &\"Filesystem\",\n\u00a0\u00a0\t... // 3 identical fields\n\u00a0\u00a0}\n" component=helm
Aug 15 12:20:57 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:57" level=info msg="I0815 12:20:57.962363 950783 topology_manager.go:215] \"Topology Admit Handler\" podUID=\"23f36f15-08fa-4b54-9def-3d565e570764\" podNamespace=\"registry\" podName=\"registry-764b9d8cc4-wm26m\"" component=kubelet stream=stderr
Aug 15 12:20:57 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:57" level=info msg="I0815 12:20:57.965505 950619 replica_set.go:676] \"Finished syncing\" kind=\"ReplicaSet\" key=\"registry/registry-764b9d8cc4\" duration=\"18.541603ms\"" component=kube-controller-manager stream=stderr
Aug 15 12:20:57 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:57" level=info msg="I0815 12:20:57.974935 950619 replica_set.go:676] \"Finished syncing\" kind=\"ReplicaSet\" key=\"registry/registry-764b9d8cc4\" duration=\"9.358792ms\"" component=kube-controller-manager stream=stderr
Aug 15 12:20:57 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:57" level=info msg="I0815 12:20:57.975057 950619 replica_set.go:676] \"Finished syncing\" kind=\"ReplicaSet\" key=\"registry/registry-764b9d8cc4\" duration=\"68.346µs\"" component=kube-controller-manager stream=stderr
Aug 15 12:20:57 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:57" level=debug msg="Upgrade failed and atomic is set, rolling back to last successful release" component=helm
Aug 15 12:20:57 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:57" level=info msg="I0815 12:20:57.977334 950619 replica_set.go:676] \"Finished syncing\" kind=\"ReplicaSet\" key=\"registry/registry-764b9d8cc4\" duration=\"96.994µs\"" component=kube-controller-manager stream=stderr
Aug 15 12:20:57 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:57" level=debug msg="getting history for release docker-registry" component=helm
Aug 15 12:20:57 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:57" level=debug msg="preparing rollback of docker-registry" component=helm
...
Aug 15 12:20:58 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:20:58" level=error msg="Reconciler error" Chart="{k0s-addon-chart-docker-registry kube-system}" component=extensions_controller controller=chart controllerGroup=helm.k0sproject.io controllerKind=Chart error="can't update or install chart: can't reconcile upgrade for \"k0s-addon-chart-docker-registry\": can't upgrade loadedChart `docker-registry`: an error occurred while rolling back the release. original upgrade error: failed to replace object: PersistentVolumeClaim \"registry\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims\n\u00a0\u00a0core.PersistentVolumeClaimSpec{\n\u00a0\u00a0\tAccessModes: {\"ReadWriteOnce\"},\n\u00a0\u00a0\tSelector: nil,\n\u00a0\u00a0\tResources: {Requests: {s\"storage\": {i: {...}, s: \"10Gi\", Format: \"BinarySI\"}}},\n-\u00a0\tVolumeName: \"pvc-cadec2a9-d4f7-441d-bf44-0a2925bc8a2e\",\n+\u00a0\tVolumeName: \"\",\n\u00a0\u00a0\tStorageClassName: &\"openebs-hostpath\",\n\u00a0\u00a0\tVolumeMode: &\"Filesystem\",\n\u00a0\u00a0\t... // 3 identical fields\n\u00a0\u00a0}\n: failed to replace object: PersistentVolumeClaim \"registry\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims\n\u00a0\u00a0core.PersistentVolumeClaimSpec{\n\u00a0\u00a0\tAccessModes: {\"ReadWriteOnce\"},\n\u00a0\u00a0\tSelector: nil,\n\u00a0\u00a0\tResources: {Requests: {s\"storage\": {i: {...}, s: \"10Gi\", Format: \"BinarySI\"}}},\n-\u00a0\tVolumeName: \"pvc-cadec2a9-d4f7-441d-bf44-0a2925bc8a2e\",\n+\u00a0\tVolumeName: \"\",\n\u00a0\u00a0\tStorageClassName: &\"openebs-hostpath\",\n\u00a0\u00a0\tVolumeMode: &\"Filesystem\",\n\u00a0\u00a0\t... // 3 identical fields\n\u00a0\u00a0}\n" name=k0s-addon-chart-docker-registry namespace=kube-system reconcileID="\"78a8400b-d809-4f1c-91ce-4a297559b31b\""
...
Aug 15 12:22:52 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:22:52" level=debug msg="Installed or updated reconciliation request: kube-system/k0s-addon-chart-goldpinger" component=extensions_controller extensions_type=helm
Aug 15 12:22:52 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:22:52" level=debug msg="Install or update reconciliation request: kube-system/k0s-addon-chart-docker-registry" component=extensions_controller extensions_type=helm
Aug 15 12:22:52 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:22:52" level=debug msg="Installed or updated reconciliation request: kube-system/k0s-addon-chart-docker-registry" component=extensions_controller extensions_type=helm
Aug 15 12:22:52 ethanm-ec-1 k0s[950562]: time="2024-08-15 12:22:52" level=info msg="I0815 12:22:52.882974 950783 kubelet_volumes.go:163] \"Cleaned up orphaned pod volumes dir\" podUID=\"bfeecbcc-3acd-46f1-a7fa-4a00a7c82469\" path=\"/var/lib/k0s/kubelet/pods/bfeecbcc-3acd-46f1-a7fa-4a00a7c82469/volumes\"" component=kubelet stream=stderr
Steps to reproduce
install helm
install k0s and csi
k0s install controller --single
k0s start
k0s kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.28/deploy/local-path-storage.yaml
install chart
helm repo add twuni https://helm.twun.io
helm pull twuni/docker-registry --version 2.2.3
root@ethanm-ec-1:/home/ethan# k0s kubectl apply -f k0schart.yaml
chart.helm.k0sproject.io/k0s-addon-chart-docker-registry created
root@ethanm-ec-1:/home/ethan# helm history -n registry docker-registry
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/lib/k0s/pki/admin.conf
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Thu Aug 15 20:23:26 2024 deployed docker-registry-2.2.3 2.8.1 Install complete
upgrade chart
root@ethanm-ec-1:/home/ethan# k0s kubectl apply -f upk0schart.yaml
chart.helm.k0sproject.io/k0s-addon-chart-docker-registry configured
root@ethanm-ec-1:/home/ethan# helm history -n registry docker-registry
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/lib/k0s/pki/admin.conf
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Thu Aug 15 20:23:26 2024 deployed docker-registry-2.2.3 2.8.1 Install complete
2 Thu Aug 15 20:27:39 2024 superseded docker-registry-2.2.3 2.8.1 Upgrade "docker-registry" failed: failed to replace object: PersistentVolumeClaim "docker-registry" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound clai...
3 Thu Aug 15 20:27:40 2024 failed docker-registry-2.2.3 2.8.1 Rollback "docker-registry" failed: failed to replace object: PersistentVolumeClaim "docker-registry" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound cla...
See the error in the logs
Aug 15 20:27:40 ethanm-ec-1 k0s[1261741]: time="2024-08-15 20:27:40" level=info msg="E0815 20:27:40.205184 1261898 pod_workers.go:1298] \"Error syncing pod, skipping\" err=\"unmounted volumes=[data kube-api-access-q58bk], unattached volumes=[], failed to process volumes=[]: context canceled\" pod=\"registry/docker-registry-8485d69f49-r8gzj\" podUID=\"485fd833-85be-47e0-a47f-7b9ea7f46a2a\"" component=kubelet stream=stderr
Aug 15 20:27:40 ethanm-ec-1 k0s[1261741]: time="2024-08-15 20:27:40" level=info msg="Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes reqeueuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler" Chart="{k0s-addon-chart-docker-registry kube-system}" component=extensions_controller controller=chart controllerGroup=helm.k0sproject.io controllerKind=Chart name=k0s-addon-chart-docker-registry namespace=kube-system reconcileID="\"43d283ce-58e3-4356-9bd4-fab9c5301bd6\""
Aug 15 20:27:40 ethanm-ec-1 k0s[1261741]: time="2024-08-15 20:27:40" level=error msg="Reconciler error" Chart="{k0s-addon-chart-docker-registry kube-system}" component=extensions_controller controller=chart controllerGroup=helm.k0sproject.io controllerKind=Chart error="can't update or install chart: can't reconcile upgrade for \"k0s-addon-chart-docker-registry\": can't upgrade loadedChart `docker-registry`: an error occurred while rolling back the release. original upgrade error: failed to replace object: PersistentVolumeClaim \"docker-registry\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims\n core.PersistentVolumeClaimSpec{\n \tAccessModes: {\"ReadWriteOnce\"},\n \tSelector: nil,\n \tResources: {Requests: {s\"storage\": {i: {...}, s: \"10Gi\", Format: \"BinarySI\"}}},\n- \tVolumeName: \"pvc-3e8cf506-0a8b-438e-9fb1-4bd7b371d0ae\",\n+ \tVolumeName: \"\",\n \tStorageClassName: &\"local-path\",\n \tVolumeMode: &\"Filesystem\",\n \t... // 3 identical fields\n }\n: failed to replace object: PersistentVolumeClaim \"docker-registry\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims\n core.PersistentVolumeClaimSpec{\n \tAccessModes: {\"ReadWriteOnce\"},\n \tSelector: nil,\n \tResources: {Requests: {s\"storage\": {i: {...}, s: \"10Gi\", Format: \"BinarySI\"}}},\n- \tVolumeName: \"pvc-3e8cf506-0a8b-438e-9fb1-4bd7b371d0ae\",\n+ \tVolumeName: \"\",\n \tStorageClassName: &\"local-path\",\n \tVolumeMode: &\"Filesystem\",\n \t... // 3 identical fields\n }\n" name=k0s-addon-chart-docker-registry namespace=kube-system reconcileID="\"43d283ce-58e3-4356-9bd4-fab9c5301bd6\""
Aug 15 20:27:40 ethanm-ec-1 k0s[1261741]: time="2024-08-15 20:27:40" level=info msg="I0815 20:27:40.213714 1261822 replica_set.go:676] \"Finished syncing\" logger=\"replicaset-controller\" kind=\"ReplicaSet\" key=\"registry/docker-registry-8485d69f49\" duration=\"15.477802ms\"" component=kube-controller-manager stream=stderr
There is nothing in the status
root@ethanm-ec-1:/home/ethan# k0s kubectl -n kube-system get charts -oyaml k0s-addon-chart-docker-registry
apiVersion: helm.k0sproject.io/v1beta1
kind: Chart
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"helm.k0sproject.io/v1beta1","kind":"Chart","metadata":{"annotations":{},"name":"k0s-addon-chart-docker-registry","namespace":"kube-system"},"spec":{"chartName":"/home/ethan/docker-registry-2.2.3.tgz","namespace":"registry","releaseName":"docker-registry","timeout":"0s","values":"image:\n tag: 2.8.3\npersistence:\n accessMode: ReadWriteOnce\n enabled: true\n size: 10Gi\n storageClass: local-path\nreplicaCount: 1\nstorage: filesystem\nextraEnvVars: []\n - name: TESTING\n value: \"testing\"\n","version":"2.2.3"}}
creationTimestamp: "2024-08-15T20:23:26Z"
generation: 2
name: k0s-addon-chart-docker-registry
namespace: kube-system
resourceVersion: "908"
uid: aa17b9e8-9498-47e1-8824-5f9c65283bf9
spec:
chartName: /home/ethan/docker-registry-2.2.3.tgz
namespace: registry
releaseName: docker-registry
timeout: 0s
values: |
image:
tag: 2.8.3
persistence:
accessMode: ReadWriteOnce
enabled: true
size: 10Gi
storageClass: local-path
replicaCount: 1
storage: filesystem
extraEnvVars: []
- name: TESTING
value: "testing"
version: 2.2.3
status:
appVersion: 2.8.1
namespace: registry
releaseName: docker-registry
revision: 1
updated: 2024-08-15 20:27:40.215240088 +0000 UTC m=+495.056953567
valuesHash: f749a71a7f93817862fe979e7fffba021f591ebf8b5bd29d5e2fe23f67b6eecb
version: 2.2.3
root@ethanm-ec-1:/home/ethan# cat k0schart.yaml
apiVersion: helm.k0sproject.io/v1beta1
kind: Chart
metadata:
name: k0s-addon-chart-docker-registry
namespace: kube-system
spec:
chartName: /home/ethan/docker-registry-2.2.3.tgz
namespace: registry
releaseName: docker-registry
timeout: 0s
values: |2
image:
tag: 2.8.3
persistence:
accessMode: ReadWriteOnce
enabled: true
size: 10Gi
storageClass: local-path
replicaCount: 1
storage: filesystem
version: 2.2.3
root@ethanm-ec-1:/home/ethan# cat upk0schart.yaml
apiVersion: helm.k0sproject.io/v1beta1
kind: Chart
metadata:
name: k0s-addon-chart-docker-registry
namespace: kube-system
spec:
chartName: /home/ethan/docker-registry-2.2.3.tgz
namespace: registry
releaseName: docker-registry
timeout: 0s
values: |2
image:
tag: 2.8.3
persistence:
accessMode: ReadWriteOnce
enabled: true
size: 10Gi
storageClass: local-path
replicaCount: 1
storage: filesystem
extraEnvVars: []
- name: TESTING
value: "testing"
version: 2.2.3
Expected behavior
The status shows the error
Actual behavior
The status does not show the error
Screenshots and logs
No response
Additional context
No response
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment