Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block

[slieer]

Describe the bug

Rocky Linux release 8.10 (Green Obsidian)
minikube version: v1.34.0
kubectl version: Client Version: v1.31.2 Kustomize Version: v5.4.2 Server Version: v1.31.0
chectl version:
chectl/7.94.0 linux-x64 node-v18.18.0 or chectl/7.93.0 linux-x64 node-v18.18.0

chectl server:deploy --platform minikube
› Current Kubernetes context: 'minikube'
✔ Verify Kubernetes API...[1.31]
✔ Minikube preflight checklist
✔ Verify if kubectl is installed...[OK]
✔ Verify if minikube is installed...[OK]
✔ Verify if minikube is running...[OK]
✔ Enable minikube ingress addon...[Enabled]
✔ Retrieving minikube IP and domain for ingress URLs...[192.168.49.2.nip.io]
✔ Checking minikube version...[1.34.0]
✔ Create Namespace eclipse-che...[Exists]
✔ Install Cert Manager v1.8.2
✔ Apply resources...[Exists]
✔ Wait for Cert Manager pods ready...[OK]
✔ Install Dex
✔ Create Namespace dex...[Exists]
✔ Create Certificates...[Exists: /tmp/dex-ca.crt]
✔ Create ConfigMap dex-ca...[Updated]
✔ Create ServiceAccount dex...[Exists]
✔ Create ClusterRole dex...[Exists]
✔ Create ClusterRoleBinding dex...[Exists]
✔ Create Service dex...[Exists]
✔ Create Ingress dex...[Exists]
✔ Generate Dex username and password...[Exists]
✔ Create ConfigMap dex...[Exists]
✔ Create Deployment dex...[Exists]
✔ Configure API server
✔ Create /etc/ca-certificates directory...[Created]
✔ Copy Dex certificate into Minikube...[OK]
✔ Configure Minikube API server...[OK]
✔ Wait for Minikube API server...[OK]
✔ Start following Eclipse Che installation logs...[OK]
❯ Deploy Eclipse Che operator
❯ Install Dev Workspace operator
✔ Create Namespace devworkspace-controller...[Exists]
✖ Create Dev Workspace operator resources
→ issuer.cert-manager.io/devworkspace-controller-selfsigned-issuer unchanged
Wait for Dev Workspace operator ready
Create ServiceAccount che-operator
Create RBAC
Wait for Cert Manager pods ready
Create Certificate che-operator-serving-cert
Create Issuer che-operator-selfsigned-issuer
Create Service che-operator-service
Create CRD checlusters.org.eclipse.che
Waiting
Create Deployment che-operator
Eclipse Che Operator pod bootstrap
Create ValidatingWebhookConfiguration org.eclipse.che
Create MutatingWebhookConfiguration org.eclipse.che
Create CheCluster Custom Resource
Error: Command server:deploy failed with the error: Command failed with exit code 1: kubectl apply -f /usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml
Error from server (Invalid): error when applying patch:
{"spec":{"conversion":{"webhook":{"clientConfig":{"caBundle":"Cg=="}}}},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}}
to:
Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "devworkspaces.workspace.devfile.io", Namespace: ""
for: "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": error when patching "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": CustomResourceDefinition.apiextensions.k8s.io "devworkspaces.workspace.devfile.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block
Error from server (Invalid): error when applying patch:
{"spec":{"conversion":{"webhook":{"clientConfig":{"caBundle":"Cg=="}}}},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}}
to:
Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "devworkspacetemplates.workspace.devfile.io", Namespace: ""
for: "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": error when patching "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": CustomResourceDefinition.apiextensions.k8s.io "devworkspacetemplates.workspace.devfile.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block
customresourcedefinition.apiextensions.k8s.io/devworkspaceoperatorconfigs.controller.devfile.io configured
customresourcedefinition.apiextensions.k8s.io/devworkspaceroutings.controller.devfile.io configured
serviceaccount/devworkspace-controller-serviceaccount unchanged
role.rbac.authorization.k8s.io/devworkspace-controller-leader-election-role unchanged
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-edit-workspaces unchanged
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-metrics-reader unchanged
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-proxy-role unchanged
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-role configured
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-view-workspaces unchanged
rolebinding.rbac.authorization.k8s.io/devworkspace-controller-leader-election-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/devworkspace-controller-proxy-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/devworkspace-controller-rolebinding unchanged
service/devworkspace-controller-manager-service unchanged
service/devworkspace-controller-metrics unchanged
deployment.apps/devworkspace-controller-manager configured
certificate.cert-manager.io/devworkspace-controller-serving-cert unchanged
issuer.cert-manager.io/devworkspace-controller-selfsigned-issuer unchanged See details: /home/skyworth/.cache/chectl/error.log. Eclipse Che logs: /tmp/chectl-logs/1730961011828.
at newError (/usr/local/lib/chectl/lib/utils/utls.js:39:19)
at wrapCommandError (/usr/local/lib/chectl/lib/utils/command-utils.js:54:32)
at Deploy. (/usr/local/lib/chectl/lib/commands/server/deploy.js:82:65)
at Generator.throw ()
at rejected (/usr/local/lib/chectl/node_modules/tslib/tslib.js:167:69)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Cause: Error: Command failed with exit code 1: kubectl apply -f /usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml
Error from server (Invalid): error when applying patch:
{"spec":{"conversion":{"webhook":{"clientConfig":{"caBundle":"Cg=="}}}},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}}
to:
Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "devworkspaces.workspace.devfile.io", Namespace: ""
for: "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": error when patching "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": CustomResourceDefinition.apiextensions.k8s.io "devworkspaces.workspace.devfile.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block
Error from server (Invalid): error when applying patch:
{"spec":{"conversion":{"webhook":{"clientConfig":{"caBundle":"Cg=="}}}},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}}
to:
Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "devworkspacetemplates.workspace.devfile.io", Namespace: ""
for: "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": error when patching "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": CustomResourceDefinition.apiextensions.k8s.io "devworkspacetemplates.workspace.devfile.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block
customresourcedefinition.apiextensions.k8s.io/devworkspaceoperatorconfigs.controller.devfile.io configured
customresourcedefinition.apiextensions.k8s.io/devworkspaceroutings.controller.devfile.io configured
serviceaccount/devworkspace-controller-serviceaccount unchanged
role.rbac.authorization.k8s.io/devworkspace-controller-leader-election-role unchanged
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-edit-workspaces unchanged
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-metrics-reader unchanged
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-proxy-role unchanged
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-role configured
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-view-workspaces unchanged
rolebinding.rbac.authorization.k8s.io/devworkspace-controller-leader-election-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/devworkspace-controller-proxy-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/devworkspace-controller-rolebinding unchanged
service/devworkspace-controller-manager-service unchanged
service/devworkspace-controller-metrics unchanged
deployment.apps/devworkspace-controller-manager configured
certificate.cert-manager.io/devworkspace-controller-serving-cert unchanged
issuer.cert-manager.io/devworkspace-controller-selfsigned-issuer unchanged
at makeError (/usr/local/lib/chectl/node_modules/execa/lib/error.js:60:11)
at handlePromise (/usr/local/lib/chectl/node_modules/execa/index.js:118:26)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

minikube kubectl -- get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
cert-manager cert-manager-54f9d599b-mn52s 1/1 Running 4 (4m28s ago) 44h
cert-manager cert-manager-cainjector-648f59958c-ws8nk 1/1 Running 6 (4m29s ago) 44h
cert-manager cert-manager-webhook-7b845b56cb-k9gdj 1/1 Running 5 (4m29s ago) 44h
devworkspace-controller devworkspace-controller-manager-f54dbb6f6-vs55l 2/2 Running 0 3m12s
devworkspace-controller devworkspace-webhook-server-7c4b65bdb9-2t9nt 2/2 Running 0 118s
devworkspace-controller devworkspace-webhook-server-7c4b65bdb9-kt9lf 2/2 Running 0 2m18s
dex dex-7687bb6d68-k68gc 1/1 Running 7 (3m57s ago) 44h
ingress-nginx ingress-nginx-admission-create-btgxv 0/1 Completed 0 45h
ingress-nginx ingress-nginx-admission-patch-46wbc 0/1 Completed 0 45h
ingress-nginx ingress-nginx-controller-857f8876df-dn89f 1/1 Running 4 (4m19s ago) 45h
kube-system coredns-d4ddbc888-72f9c 1/1 Running 5 (4m24s ago) 46h
kube-system coredns-d4ddbc888-xn45q 1/1 Running 4 (4m24s ago) 46h
kube-system etcd-minikube 1/1 Running 6 (4m28s ago) 46h
kube-system kube-apiserver-minikube 1/1 Running 3 (4m18s ago) 44h
kube-system kube-controller-manager-minikube 1/1 Running 5 (4m29s ago) 46h
kube-system kube-proxy-xqvwd 1/1 Running 5 (4m29s ago) 46h
kube-system kube-scheduler-minikube 1/1 Running 5 (4m28s ago) 46h
kube-system metrics-server-686dff4775-j2dhq 1/1 Running 8 (3m57s ago) 45h
kube-system storage-provisioner 1/1 Running 6 (4m29s ago) 46h
kubernetes-dashboard dashboard-metrics-scraper-c5db448b4-jdmwx 1/1 Running 4 (4m29s ago) 45h
kubernetes-dashboard kubernetes-dashboard-695b96c756-qfrdq 1/1 Running 5 (4m28s ago) 45h

Che version

7.93/ 7.94

Steps to reproduce

chectl server:deploy --platform minikube

Expected behavior

che install success.

Runtime

minikube

Screenshots

No response

Installation method

chectl/latest

Environment

Rocky Linux release 8.10 (Green Obsidian)

Eclipse Che Logs

No response

Additional context

No response

[tolusha]

It seems it is impossible to deploy DWO/Che operator on the latest Kubernetes version

[tolusha]

@slieer
Could reinstall minikube and deploy che one more time?
Currently I can't reproduce the issue.
Sometimes I have storage is (re)initializing problem. Maybe the latest minikube is not stable.

[slieer]

@slieer Could reinstall minikube and deploy che one more time? Currently I can't reproduce the issue. Sometimes I have storage is (re)initializing problem. Maybe the latest minikube is not stable.

Thank you for your response. I'll try again.