Admission webhooks and CSVs

[ukclivecox]

I see there is a proposal to extend the CSV to add admission webhooks.

Is there any update on this?

What is the workaround for now? Is it assume the operator itself will create any resources not currently allowed in the CSV at startup?

We use kubebuilder and these admission webhooks are defined in yaml rather than dynamically created.

[ecordell]

We are planning to ship admission webhook support within the coming weeks (probably 3-4 weeks out)

The workaround for now is to have your operator create the admission webhook itself, but we don't recommend that, since the certs you configure will not be rotated by default (and eventually will expire, and your webhook will start failing)

[raffaelespazzoli]

I also would like to see this feature.

[dmesser]

The next upstream version of OLM will have it, the implementation just reached code-complete.

Design: https://github.com/operator-framework/operator-lifecycle-manager/blob/master/doc/contributors/design-proposals/webhooks.md

[kfox1111]

Been playing around with this. It worked in my minikube environment (1.18). When bringing it over to a production cluster (1.16) the webhooks are failing to deploy properly. Is there some feature (Maybe ca creation) that requires a newer version of k8s?

The mutatingwebhook configuration is created properly. The deployment goes through, but its missing the secret with the certs in it.

The logs are spewing constantly in a loop:

I0605 16:42:59.002104       1 event.go:278] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"xxxx", Name:"xxxxmutator.v0.1.0", UID:"2a20b3db-7c47-45b5-a3a1-22348d503a2f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"135737167", FieldPath:""}): type: 'Normal' reason: 'InstallSucceeded' waiting for install components to report healthy
time="2020-06-05T16:42:59Z" level=info msg="error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"xxxxmutator.v0.1.0\": the object has been modified; please apply your changes to the latest version and try again" csv=xxxxmutator.v0.1.0 id=RK/8z namespace=xxxx phase=InstallReady
E0605 16:42:59.010506       1 queueinformer_operator.go:290] sync {"update" "xxxx/xxxxmutator.v0.1.0"} failed: error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com "xxxxmutator.v0.1.0": the object has been modified; please apply your changes to the latest version and try again
time="2020-06-05T16:42:59Z" level=info msg="csv in operatorgroup" csv=xxxxmutator.v0.1.0 id=A6wqB namespace=xxxx opgroup=xxxx phase=Installing
I0605 16:42:59.051509       1 event.go:278] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"xxxx", Name:"xxxxmutator.v0.1.0", UID:"2a20b3db-7c47-45b5-a3a1-22348d503a2f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"135737175", FieldPath:""}): type: 'Normal' reason: 'NeedsReinstall' calculated deployment install is bad
time="2020-06-05T16:42:59Z" level=info msg="csv in operatorgroup" csv=xxxxmutator.v0.1.0 id=Ty5iR namespace=xxxx opgroup=xxxx phase=Installing
I0605 16:42:59.101739       1 event.go:278] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"xxxx", Name:"xxxxmutator.v0.1.0", UID:"2a20b3db-7c47-45b5-a3a1-22348d503a2f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"135737175", FieldPath:""}): type: 'Normal' reason: 'NeedsReinstall' calculated deployment install is bad
time="2020-06-05T16:42:59Z" level=info msg="error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"xxxxmutator.v0.1.0\": the object has been modified; please apply your changes to the latest version and try again" csv=xxxxmutator.v0.1.0 id=2PUay namespace=xxxx phase=Installing
E0605 16:42:59.109681       1 queueinformer_operator.go:290] sync {"update" "xxxx/xxxxmutator.v0.1.0"} failed: error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com "xxxxmutator.v0.1.0": the object has been modified; please apply your changes to the latest version and try again
time="2020-06-05T16:42:59Z" level=info msg="csv in operatorgroup" csv=xxxxmutator.v0.1.0 id=B3r7T namespace=xxxx opgroup=xxxx phase=Pending
time="2020-06-05T16:42:59Z" level=info msg="scheduling ClusterServiceVersion for install" csv=xxxxmutator.v0.1.0 id=B3r7T namespace=xxxx phase=Pending
I0605 16:42:59.110114       1 event.go:278] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"xxxx", Name:"xxxxmutator.v0.1.0", UID:"2a20b3db-7c47-45b5-a3a1-22348d503a2f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"135737180", FieldPath:""}): type: 'Normal' reason: 'AllRequirementsMet' all requirements found, attempting install
time="2020-06-05T16:42:59Z" level=info msg="csv in operatorgroup" csv=xxxxmutator.v0.1.0 id=/6aYN namespace=xxxx opgroup=xxxx phase=Pending
time="2020-06-05T16:42:59Z" level=info msg="scheduling ClusterServiceVersion for install" csv=xxxxmutator.v0.1.0 id=/6aYN namespace=xxxx phase=Pending
I0605 16:42:59.123581       1 event.go:278] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"xxxx", Name:"xxxxmutator.v0.1.0", UID:"2a20b3db-7c47-45b5-a3a1-22348d503a2f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"135737180", FieldPath:""}): type: 'Normal' reason: 'AllRequirementsMet' all requirements found, attempting install
time="2020-06-05T16:42:59Z" level=info msg="error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"xxxxmutator.v0.1.0\": the object has been modified; please apply your changes to the latest version and try again" csv=xxxxmutator.v0.1.0 id=5Dq/j namespace=xxxx phase=Pending
E0605 16:42:59.130999       1 queueinformer_operator.go:290] sync {"update" "xxxx/xxxxmutator.v0.1.0"} failed: error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com "xxxxmutator.v0.1.0": the object has been modified; please apply your changes to the latest version and try again
time="2020-06-05T16:42:59Z" level=info msg="csv in operatorgroup" csv=xxxxmutator.v0.1.0 id=LJGkp namespace=xxxx opgroup=xxxx phase=InstallReady
time="2020-06-05T16:42:59Z" level=info msg="No api or webhook descs to add CA to"

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[rojkov]

I'm developing an operator based on Kubebuilder and this feature of OLM puzzles me.

After a long search I finally stumbled upon this doc which says that "The tls Cert file will be mounted to the deployment at /apiserver.local.config/certificates/apiserver.crt". But controller-runtime expects all webhooks to be protected with certificates and keys named respectively tls.crt and tls.key.

Yet the blog post about operator-sdk reaching v1.0 insists that "existing Kubebuilder projects can be used as is with the SDK and continue to work" and "for Golang Operators the SDK supports webhooks to be created via Kubebuilder and managed by OLM".

So, I can modify my operator to specify explicitly CertDir compatible with OLM, but I can't find a way to do the same with file names inside a Secret. As result my operator fails to find the webhook keys when deployed with OLM.

[awgreene]

Hello @rojkov,

Thanks for creating this issue. As you said, OLM currently mounts the CA files at a different location than the controller-runtime expects. We will adress this issue soon in order to better align with controller-runtime, but in the meantime you can deploy your operator with OLM by repeating the steps in this commit which changes where the operator searches for the certs:

const (
	WebhookPort     = 4343
	WebhookCertDir  = "/apiserver.local.config/certificates"
	WebhookCertName = "apiserver.crt"
	WebhookKeyName  = "apiserver.key"
)

func (r *WebhookTest) SetupWebhookWithManager(mgr ctrl.Manager) error {
	bldr := ctrl.NewWebhookManagedBy(mgr).
		For(r)

	// Specify OLM CA Info
	srv := mgr.GetWebhookServer()
	srv.CertDir = WebhookCertDir
	srv.CertName = WebhookCertName
	srv.KeyName = WebhookKeyName
	srv.Port = WebhookPort

	return bldr.Complete()
}

[rojkov]

@awgreene Thanks! That helps.

Now I've got the same problem as @kfox1111 reported: there is no secret mounted to the operator's containers with the needed keys. In the logs I see the same messages like

time="2020-06-05T16:42:59Z" level=info msg="No api or webhook descs to add CA to"

I'm testing against K8s v1.18.2 running inside Kind

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-30T20:19:45Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

My CSV.

[awgreene]

Hello @rojkov - thanks for using OLM!

Your CSV has a single deployment named inteldeviceplugins-controller-manager - this does not match the name of the deployment in your webhookDefinitions.

OLM uses the deploymentName filed in the WebhookDefinition object to ID which deployments to inject the CA cert into. Please make sure that value provided in the webhookDefinition's deploymentName field matches the name of a deployment in you CSV.