Another failed upgrade: cannot unmarshal string into Go struct field PodSpec.spec.template.spec.imagePullSecrets of type v1.LocalObjectReference

[darkpixel]

I dread ChatWoot upgrades.
I haven't had a clean upgrade since we started using it.
There's always something being changed in the chart that breaks things.

This time it's something about imagePullSecrets.

Error: Command failed: /tmp/.mount_Lens-2EH4iuK/resources/x64/helm upgrade caitwoot chatwoot/chatwoot --version 1.1.8 --values /tmp/14d923a3-b9c6-43fd-b3ad-aa1326fd1162-values.yaml --namespace woot --kubeconfig /tmp/kubeconfig-c98c68b5fd67d453a90b0076edf39f54
Error: UPGRADE FAILED: cannot patch "caitwoot-chatwoot-web" with kind Deployment: "" is invalid: patch: Invalid value: "{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"deployment.kubernetes.io/revision\":\"20\",\"meta.helm.sh/release-name\":\"caitwoot\",\"meta.helm.sh/release-namespace\":\"woot\"},\"creationTimestamp\":\"2022-12-21T16:11:08Z\",\"generation\":21,\"labels\":{\"app\":\"caitwoot-chatwoot\",\"app.kubernetes.io/managed-by\":\"Helm\",\"chart\":\"chatwoot-1.1.8\",\"heritage\":\"Helm\",\"release\":\"caitwoot\"},\"managedFields\":[{\"manager\":\"node-fetch\",\"operation\":\"Update\",\"apiVersion\":\"apps/v1\",\"time\":\"2023-02-13T15:08:34Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:spec\":{\"f:template\":{\"f:metadata\":{\"f:annotations\":{\"f:kubectl.kubernetes.io/restartedAt\":{}}}}}}},{\"manager\":\"helm\",\"operation\":\"Update\",\"apiVersion\":\"apps/v1\",\"time\":\"2024-02-28T14:58:25Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:meta.helm.sh/release-name\":{},\"f:meta.helm.sh/release-namespace\":{}},\"f:labels\":{\".\":{},\"f:app\":{},\"f:app.kubernetes.io/managed-by\":{},\"f:chart\":{},\"f:heritage\":{},\"f:release\":{}}},\"f:spec\":{\"f:progressDeadlineSeconds\":{},\"f:replicas\":{},\"f:revisionHistoryLimit\":{},\"f:selector\":{},\"f:strategy\":{\"f:rollingUpdate\":{\".\":{},\"f:maxSurge\":{},\"f:maxUnavailable\":{}},\"f:type\":{}},\"f:template\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:checksum/config\":{}},\"f:labels\":{\".\":{},\"f:app\":{},\"f:release\":{},\"f:role\":{}}},\"f:spec\":{\"f:containers\":{\"k:{\\\"name\\\":\\\"chatwoot-web\\\"}\":{\".\":{},\"f:args\":{},\"f:command\":{},\"f:envFrom\":{},\"f:image\":{},\"f:imagePullPolicy\":{},\"f:name\":{},\"f:ports\":{\".\":{},\"k:{\\\"containerPort\\\":3000,\\\"protocol\\\":\\\"TCP\\\"}\":{\".\":{},\"f:containerPort\":{},\"f:protocol\":{}}},\"f:resources\":{},\"f:terminationMessagePath\":{},\"f:terminationMessagePolicy\":{},\"f:volumeMounts\":{\".\":{},\"k:{\\\"mountPath\\\":\\\"/app/tmp\\\"}\":{\".\":{},\"f:mountPath\":{},\"f:name\":{}}}}},\"f:dnsPolicy\":{},\"f:restartPolicy\":{},\"f:schedulerName\":{},\"f:securityContext\":{},\"f:serviceAccount\":{},\"f:serviceAccountName\":{},\"f:terminationGracePeriodSeconds\":{},\"f:volumes\":{\".\":{},\"k:{\\\"name\\\":\\\"cache\\\"}\":{\".\":{},\"f:emptyDir\":{},\"f:name\":{}}}}}}}},{\"manager\":\"kube-controller-manager\",\"operation\":\"Update\",\"apiVersion\":\"apps/v1\",\"time\":\"2024-03-18T17:55:50Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:deployment.kubernetes.io/revision\":{}}},\"f:status\":{\"f:availableReplicas\":{},\"f:conditions\":{\".\":{},\"k:{\\\"type\\\":\\\"Available\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:lastUpdateTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"Progressing\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:lastUpdateTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}}},\"f:observedGeneration\":{},\"f:readyReplicas\":{},\"f:replicas\":{},\"f:updatedReplicas\":{}}},\"subresource\":\"status\"}],\"name\":\"caitwoot-chatwoot-web\",\"namespace\":\"woot\",\"resourceVersion\":\"384969638\",\"uid\":\"4f2d70d5-2948-4694-8e80-18735ed1bb57\"},\"spec\":{\"progressDeadlineSeconds\":600,\"replicas\":1,\"revisionHistoryLimit\":10,\"selector\":{\"matchLabels\":{\"app\":\"caitwoot-chatwoot\",\"release\":\"caitwoot\",\"role\":\"web\"}},\"strategy\":{\"type\":\"RollingUpdate\",\"rollingUpdate\":{\"maxUnavailable\":\"25%!\(MISSING)",\"maxSurge\":\"25%!\(MISSING)"}},\"template\":{\"metadata\":{\"annotations\":{\"checksum/config\":\"e013226117bca079b0bafe2427a0acfd4ec5feb680f512ee6239ad3ca22ecd61\",\"kubectl.kubernetes.io/restartedAt\":\"2023-02-13T15:08:34Z\"},\"creationTimestamp\":null,\"labels\":{\"app\":\"caitwoot-chatwoot\",\"release\":\"caitwoot\",\"role\":\"web\"}},\"spec\":{\"containers\":[{\"args\":[\"bundle\",\"exec\",\"rails\",\"s\",\"-p\",\"3000\",\"-b\",\"0.0.0.0\"],\"command\":[\"docker/entrypoints/rails.sh\"],\"envFrom\":[{\"secretRef\":{\"name\":\"caitwoot-chatwoot-env\"}}],\"image\":\"chatwoot/chatwoot:v3.7.0\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"chatwoot-web\",\"ports\":[{\"containerPort\":3000}],\"volumeMounts\":[{\"mountPath\":\"/app/tmp\",\"name\":\"cache\"}]}],\"dnsPolicy\":\"ClusterFirst\",\"imagePullSecrets\":[\"docker-io-key\"],\"restartPolicy\":\"Always\",\"schedulerName\":\"default-scheduler\",\"securityContext\":{},\"serviceAccount\":\"caitwoot-chatwoot\",\"serviceAccountName\":\"caitwoot-chatwoot\",\"terminationGracePeriodSeconds\":30,\"volumes\":[{\"name\":\"cache\",\"emptyDir\":{}}]}}},\"status\":{\"observedGeneration\":21,\"replicas\":1,\"updatedReplicas\":1,\"readyReplicas\":1,\"availableReplicas\":1,\"conditions\":[{\"type\":\"Progressing\",\"status\":\"True\",\"lastUpdateTime\":\"2024-02-28T14:59:27Z\",\"lastTransitionTime\":\"2023-04-11T23:28:26Z\",\"reason\":\"NewReplicaSetAvailable\",\"message\":\"ReplicaSet \\\"caitwoot-chatwoot-web-6988b4897b\\\" has successfully progressed.\"},{\"type\":\"Available\",\"status\":\"True\",\"lastUpdateTime\":\"2024-03-18T17:55:50Z\",\"lastTransitionTime\":\"2024-03-18T17:55:50Z\",\"reason\":\"MinimumReplicasAvailable\",\"message\":\"Deployment has minimum availability.\"}]}}": json: cannot unmarshal string into Go struct field PodSpec.spec.template.spec.imagePullSecrets of type v1.LocalObjectReference && cannot patch "caitwoot-chatwoot-worker" with kind Deployment: "" is invalid: patch: Invalid value: "{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"deployment.kubernetes.io/revision\":\"20\",\"meta.helm.sh/release-name\":\"caitwoot\",\"meta.helm.sh/release-namespace\":\"woot\"},\"creationTimestamp\":\"2022-12-21T16:11:08Z\",\"generation\":20,\"labels\":{\"app\":\"caitwoot-chatwoot\",\"app.kubernetes.io/managed-by\":\"Helm\",\"chart\":\"chatwoot-1.1.8\",\"heritage\":\"Helm\",\"release\":\"caitwoot\"},\"managedFields\":[{\"manager\":\"node-fetch\",\"operation\":\"Update\",\"apiVersion\":\"apps/v1\",\"time\":\"2023-02-13T15:08:37Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:spec\":{\"f:template\":{\"f:metadata\":{\"f:annotations\":{\"f:kubectl.kubernetes.io/restartedAt\":{}}}}}}},{\"manager\":\"helm\",\"operation\":\"Update\",\"apiVersion\":\"apps/v1\",\"time\":\"2024-02-28T14:58:25Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:meta.helm.sh/release-name\":{},\"f:meta.helm.sh/release-namespace\":{}},\"f:labels\":{\".\":{},\"f:app\":{},\"f:app.kubernetes.io/managed-by\":{},\"f:chart\":{},\"f:heritage\":{},\"f:release\":{}}},\"f:spec\":{\"f:progressDeadlineSeconds\":{},\"f:replicas\":{},\"f:revisionHistoryLimit\":{},\"f:selector\":{},\"f:strategy\":{\"f:rollingUpdate\":{\".\":{},\"f:maxSurge\":{},\"f:maxUnavailable\":{}},\"f:type\":{}},\"f:template\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:checksum/config\":{}},\"f:labels\":{\".\":{},\"f:app\":{},\"f:release\":{},\"f:role\":{}}},\"f:spec\":{\"f:containers\":{\"k:{\\\"name\\\":\\\"chatwoot-workers\\\"}\":{\".\":{},\"f:args\":{},\"f:envFrom\":{},\"f:image\":{},\"f:imagePullPolicy\":{},\"f:name\":{},\"f:resources\":{},\"f:terminationMessagePath\":{},\"f:terminationMessagePolicy\":{},\"f:volumeMounts\":{\".\":{},\"k:{\\\"mountPath\\\":\\\"/app/tmp\\\"}\":{\".\":{},\"f:mountPath\":{},\"f:name\":{}}}}},\"f:dnsPolicy\":{},\"f:restartPolicy\":{},\"f:schedulerName\":{},\"f:securityContext\":{},\"f:serviceAccount\":{},\"f:serviceAccountName\":{},\"f:terminationGracePeriodSeconds\":{},\"f:volumes\":{\".\":{},\"k:{\\\"name\\\":\\\"cache\\\"}\":{\".\":{},\"f:emptyDir\":{},\"f:name\":{}}}}}}}},{\"manager\":\"kube-controller-manager\",\"operation\":\"Update\",\"apiVersion\":\"apps/v1\",\"time\":\"2024-03-18T18:02:27Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:deployment.kubernetes.io/revision\":{}}},\"f:status\":{\"f:availableReplicas\":{},\"f:conditions\":{\".\":{},\"k:{\\\"type\\\":\\\"Available\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:lastUpdateTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"Progressing\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:lastUpdateTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}}},\"f:observedGeneration\":{},\"f:readyReplicas\":{},\"f:replicas\":{},\"f:updatedReplicas\":{}}},\"subresource\":\"status\"}],\"name\":\"caitwoot-chatwoot-worker\",\"namespace\":\"woot\",\"resourceVersion\":\"384977169\",\"uid\":\"f8d4c4d3-b741-4a6c-a501-93ce5b47e86f\"},\"spec\":{\"progressDeadlineSeconds\":600,\"replicas\":2,\"revisionHistoryLimit\":10,\"selector\":{\"matchLabels\":{\"app\":\"caitwoot-chatwoot\",\"release\":\"caitwoot\",\"role\":\"worker\"}},\"strategy\":{\"type\":\"RollingUpdate\",\"rollingUpdate\":{\"maxUnavailable\":\"25%!\(MISSING)",\"maxSurge\":\"25%!\(MISSING)"}},\"template\":{\"metadata\":{\"annotations\":{\"checksum/config\":\"e013226117bca079b0bafe2427a0acfd4ec5feb680f512ee6239ad3ca22ecd61\",\"kubectl.kubernetes.io/restartedAt\":\"2023-02-13T15:08:37Z\"},\"creationTimestamp\":null,\"labels\":{\"app\":\"caitwoot-chatwoot\",\"release\":\"caitwoot\",\"role\":\"worker\"}},\"spec\":{\"containers\":[{\"args\":[\"bundle\",\"exec\",\"sidekiq\",\"-C\",\"config/sidekiq.yml\"],\"envFrom\":[{\"secretRef\":{\"name\":\"caitwoot-chatwoot-env\"}}],\"image\":\"chatwoot/chatwoot:v3.7.0\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"chatwoot-workers\",\"volumeMounts\":[{\"mountPath\":\"/app/tmp\",\"name\":\"cache\"}]}],\"dnsPolicy\":\"ClusterFirst\",\"imagePullSecrets\":[\"docker-io-key\"],\"restartPolicy\":\"Always\",\"schedulerName\":\"default-scheduler\",\"securityContext\":{},\"serviceAccount\":\"caitwoot-chatwoot\",\"serviceAccountName\":\"caitwoot-chatwoot\",\"terminationGracePeriodSeconds\":30,\"volumes\":[{\"name\":\"cache\",\"emptyDir\":{}}]}}},\"status\":{\"observedGeneration\":20,\"replicas\":2,\"updatedReplicas\":2,\"readyReplicas\":2,\"availableReplicas\":2,\"conditions\":[{\"type\":\"Progressing\",\"status\":\"True\",\"lastUpdateTime\":\"2024-02-28T15:00:28Z\",\"lastTransitionTime\":\"2023-04-11T23:28:26Z\",\"reason\":\"NewReplicaSetAvailable\",\"message\":\"ReplicaSet \\\"caitwoot-chatwoot-worker-7ff456dc5b\\\" has successfully progressed.\"},{\"type\":\"Available\",\"status\":\"True\",\"lastUpdateTime\":\"2024-03-18T18:02:26Z\",\"lastTransitionTime\":\"2024-03-18T18:02:26Z\",\"reason\":\"MinimumReplicasAvailable\",\"message\":\"Deployment has minimum availability.\"}]}}": json: cannot unmarshal string into Go struct field PodSpec.spec.template.spec.imagePullSecrets of type v1.LocalObjectReference

I have it defined and the secret exists:

imagePullSecrets:
  - docker-io-key

Removing it allows the upgrade to continue...right into the brick wall that is docker.io rate-limiting.

Chart version 1.1.8.

[soudhaf]

For imagePullSecrets the format is

imagePullSecrets:
  - name: docker-io-key