Open
Description
What happened:
When I install lws with helm chart, there were following errors regarding cert-rotation, even though certs was ready in the end.
2025-01-15T05:41:49Z INFO setup both healthz and readyz check are finished and configured
2025-01-15T05:41:49Z INFO setup starting manager
2025-01-15T05:41:49Z INFO setup waiting for the cert generation to complete
2025-01-15T05:41:49Z INFO controller-runtime.metrics Starting metrics server
2025-01-15T05:41:49Z INFO setup disabling http/2
2025-01-15T05:41:49Z INFO starting server {"name": "health probe", "addr": "[::]:8081"}
2025-01-15T05:41:49Z INFO cert-rotation starting cert rotator controller
2025-01-15T05:41:49Z INFO Starting EventSource {"controller": "cert-rotator", "source": "kind source: *v1.Secret"}
2025-01-15T05:41:49Z INFO Starting EventSource {"controller": "cert-rotator", "source": "kind source: *unstructured.Unstructured"}
2025-01-15T05:41:49Z INFO Starting EventSource {"controller": "cert-rotator", "source": "kind source: *unstructured.Unstructured"}
2025-01-15T05:41:49Z INFO Starting Controller {"controller": "cert-rotator"}
I0115 05:41:49.551769 1 leaderelection.go:254] attempting to acquire leader lease lws-system/b8b2488c.x-k8s.io...
I0115 05:41:49.565390 1 leaderelection.go:268] successfully acquired lease lws-system/b8b2488c.x-k8s.io
2025-01-15T05:41:49Z DEBUG events lws-controller-manager-5bbf558c4-2mqpb_9468083c-d734-45bf-ae36-8304a73a30ad became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"lws-system","name":"b8b2488c.x-k8s.io","uid":"ddb4bf05-ed7b-4978-8438-3c688c7792ea","apiVersion":"coordination.k8s.io/v1","resourceVersion":"3408281"}, "reason": "LeaderElection"}
2025-01-15T05:41:49Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8443", "secure": true}
2025-01-15T05:41:49Z INFO cert-rotation refreshing CA and server certs
2025-01-15T05:41:49Z INFO Starting workers {"controller": "cert-rotator", "worker count": 1}
2025-01-15T05:41:49Z INFO cert-rotation refreshing CA and server certs
2025-01-15T05:41:49Z INFO cert-rotation server certs refreshed
2025-01-15T05:41:49Z ERROR cert-rotation could not refresh CA and server certs {"error": "Operation cannot be fulfilled on secrets \"lws-webhook-server-cert\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).refreshCertIfNeeded.func1
/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.12.0/pkg/rotator/rotator.go:329
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection
/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/wait.go:145
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff
/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/backoff.go:461
github.com/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).refreshCertIfNeeded
/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.12.0/pkg/rotator/rotator.go:357
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile
/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.12.0/pkg/rotator/rotator.go:772
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:224
2025-01-15T05:41:49Z INFO cert-rotation no cert refresh needed
2025-01-15T05:41:49Z ERROR cert-rotation secret is not well-formed, cannot update webhook configurations {"error": "Cert secret is not well-formed, missing ca.crt", "errorVerbose": "Cert secret is not well-formed, missing ca.crt\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.buildArtifactsFromSecret\n\t/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.12.0/pkg/rotator/rotator.go:508\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile\n\t/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.12.0/pkg/rotator/rotator.go:784\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"}
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile
/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.12.0/pkg/rotator/rotator.go:786
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:224
2025-01-15T05:41:49Z INFO cert-rotation no cert refresh needed
2025-01-15T05:41:49Z INFO cert-rotation Ensuring CA cert {"name": "lws-validating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "lws-validating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2025-01-15T05:41:49Z INFO cert-rotation Ensuring CA cert {"name": "lws-mutating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration", "name": "lws-mutating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration"}
2025-01-15T05:41:49Z INFO cert-rotation no cert refresh needed
2025-01-15T05:41:49Z INFO cert-rotation Ensuring CA cert {"name": "lws-validating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "lws-validating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2025-01-15T05:41:49Z INFO cert-rotation Ensuring CA cert {"name": "lws-mutating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration", "name": "lws-mutating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration"}
2025-01-15T05:41:51Z INFO cert-rotation certs are ready in /tmp/k8s-webhook-server/serving-certs
2025-01-15T05:41:51Z INFO cert-rotation CA certs are injected to webhooks
2025-01-15T05:41:51Z INFO setup certs ready
2025-01-15T05:41:51Z INFO Starting Controller {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod"}
2025-01-15T05:41:51Z INFO Starting workers {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "worker count": 1}
2025-01-15T05:41:51Z INFO Starting EventSource {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Pod"}
2025-01-15T05:41:51Z INFO Starting EventSource {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.StatefulSet"}
2025-01-15T05:41:51Z INFO Starting EventSource {"controller": "leaderworkerset", "controllerGroup": "leaderworkerset.x-k8s.io", "controllerKind": "LeaderWorkerSet", "source": "kind source: *v1.LeaderWorkerSet"}
2025-01-15T05:41:51Z INFO Starting EventSource {"controller": "leaderworkerset", "controllerGroup": "leaderworkerset.x-k8s.io", "controllerKind": "LeaderWorkerSet", "source": "kind source: *v1.StatefulSet"}
2025-01-15T05:41:51Z INFO Starting EventSource {"controller": "leaderworkerset", "controllerGroup": "leaderworkerset.x-k8s.io", "controllerKind": "LeaderWorkerSet", "source": "kind source: *v1.Service"}
2025-01-15T05:41:51Z INFO Starting EventSource {"controller": "leaderworkerset", "controllerGroup": "leaderworkerset.x-k8s.io", "controllerKind": "LeaderWorkerSet", "source": "kind source: *v1.StatefulSet"}
2025-01-15T05:41:51Z INFO Starting Controller {"controller": "leaderworkerset", "controllerGroup": "leaderworkerset.x-k8s.io", "controllerKind": "LeaderWorkerSet"}
2025-01-15T05:41:51Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "leaderworkerset.x-k8s.io/v1, Kind=LeaderWorkerSet", "path": "/mutate-leaderworkerset-x-k8s-io-v1-leaderworkerset"}
2025-01-15T05:41:51Z INFO controller-runtime.webhook Starting webhook server
2025-01-15T05:41:51Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-leaderworkerset-x-k8s-io-v1-leaderworkerset"}
2025-01-15T05:41:51Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "leaderworkerset.x-k8s.io/v1, Kind=LeaderWorkerSet", "path": "/validate-leaderworkerset-x-k8s-io-v1-leaderworkerset"}
2025-01-15T05:41:51Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-leaderworkerset-x-k8s-io-v1-leaderworkerset"}
2025-01-15T05:41:51Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "/v1, Kind=Pod", "path": "/mutate--v1-pod"}
2025-01-15T05:41:51Z INFO controller-runtime.certwatcher Updated current TLS certificate
2025-01-15T05:41:51Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate--v1-pod"}
2025-01-15T05:41:51Z INFO controller-runtime.webhook Serving webhook server {"host": "", "port": 9443}
2025-01-15T05:41:51Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "/v1, Kind=Pod", "path": "/validate--v1-pod"}
2025-01-15T05:41:51Z INFO controller-runtime.certwatcher Starting certificate watcher
2025-01-15T05:41:51Z INFO controller-runtime.webhook Registering webhook {"path": "/validate--v1-pod"}
2025-01-15T05:41:52Z INFO Starting workers {"controller": "leaderworkerset", "controllerGroup": "leaderworkerset.x-k8s.io", "controllerKind": "LeaderWorkerSet", "worker count": 1}
2025-01-15T05:47:26Z INFO cert-rotation no cert refresh needed
2025-01-15T05:47:26Z INFO cert-rotation Ensuring CA cert {"name": "lws-validating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "lws-validating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2025-01-15T05:47:26Z INFO cert-rotation Ensuring CA cert {"name": "lws-mutating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration", "name": "lws-mutating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration"}
2025-01-15T05:47:27Z INFO cert-rotation no cert refresh needed
2025-01-15T05:47:27Z INFO cert-rotation Ensuring CA cert {"name": "lws-validating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "lws-validating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2025-01-15T05:47:27Z INFO cert-rotation Ensuring CA cert {"name": "lws-mutating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration", "name": "lws-mutating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration"}
2025-01-15T05:47:27Z INFO cert-rotation no cert refresh needed
2025-01-15T05:47:27Z INFO cert-rotation Ensuring CA cert {"name": "lws-validating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "lws-validating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2025-01-15T05:47:27Z INFO cert-rotation Ensuring CA cert {"name": "lws-mutating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration", "name": "lws-mutating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration"}
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
- Install with helm and image.manager.tag set to v0.5.0
Anything else we need to know?:
Environment:
- Kubernetes version (use
kubectl version): v1.32.0 - LWS version (use
git describe --tags --dirty --always): v0.5.0 - Cloud provider or hardware configuration:
- OS (e.g:
cat /etc/os-release): - Kernel (e.g.
uname -a): - Install tools:
- Others: