Pipeline webhook controllers fail to become leaders

[afrittoli]

Expected Behavior

The webhook controllers become leaders and start to work

Actual Behavior

They attempt to acquire a lease, but fail.
This does not always happen, but I see it a lot in local kind clusters.

I1116 14:51:14.245992       1 leaderelection.go:242] attempting to acquire leader lease  tekton-pipelines/webhook.conversionwebhook.00-of-01...
I1116 14:51:14.246485       1 leaderelection.go:242] attempting to acquire leader lease  tekton-pipelines/webhook.webhookcertificates.00-of-01...
I1116 14:51:14.246931       1 leaderelection.go:242] attempting to acquire leader lease  tekton-pipelines/webhook.defaultingwebhook.00-of-01...
I1116 14:51:14.247191       1 leaderelection.go:242] attempting to acquire leader lease  tekton-pipelines/webhook.validationwebhook.00-of-01...
I1116 14:51:14.247479       1 leaderelection.go:242] attempting to acquire leader lease  tekton-pipelines/webhook.configmapwebhook.00-of-01...
2020/11/16 14:51:22 http: TLS handshake error from 172.18.0.4:41288: server key missing

Additional Info

The issue manifests itself through defaults missing in tekton resources.
This is caused by the webhooks not being active.
The certificate webhook does not provision the certificate, so the logs are also full of

2020/11/16 15:43:34 http: TLS handshake error from 172.18.0.4:18085: server key missing
2020/11/16 15:43:35 http: TLS handshake error from 172.18.0.4:44900: server key missing
2020/11/16 15:43:35 http: TLS handshake error from 172.18.0.4:14526: server key missing
2020/11/16 15:43:36 http: TLS handshake error from 172.18.0.4:11806: server key missing
2020/11/16 15:43:36 http: TLS handshake error from 172.18.0.4:56615: server key missing
2020/11/16 15:43:36 http: TLS handshake error from 172.18.0.4:14155: server key missing
2020/11/16 15:43:36 http: TLS handshake error from 172.18.0.4:46752: server key missing
2020/11/16 15:43:37 http: TLS handshake error from 172.18.0.4:30265: server key missing
2020/11/16 15:43:39 http: TLS handshake error from 172.18.0.4:34698: server key missing
2020/11/16 15:43:39 http: TLS handshake error from 172.18.0.4:42270: server key missing
2020/11/16 15:43:40 http: TLS handshake error from 172.18.0.4:27379: server key missing
2020/11/16 15:43:40 http: TLS handshake error from 172.18.0.4:44972: server key missing
2020/11/16 15:43:40 http: TLS handshake error from 172.18.0.4:18143: server key missing

This is caused by a naming conflict on the leases between pipeline and triggers:

$ k get leases -n tekton-pipelines
controller.github.com-tektoncd-triggers-pkg-reconciler-v1alpha1-eventlistener.reconciler.00-of-01   tekton-triggers-controller-594c47b76-mjn67_595148b8-c0fa-4084-ac36-b38d7e29a795     2d6h
tekton.github.com-tektoncd-pipeline-pkg-reconciler-pipelinerun.reconciler.00-of-01                  tekton-pipelines-controller-5f94d94f74-tvbzs_144bd503-7f9d-4e47-8c61-c0283a276632   2d6h
tekton.github.com-tektoncd-pipeline-pkg-reconciler-taskrun.reconciler.00-of-01                      tekton-pipelines-controller-5f94d94f74-tvbzs_511b61e9-09f6-462c-b6af-c17afea38735   2d6h
webhook.configmapwebhook.00-of-01                                                                   tekton-triggers-webhook-59df6955c8-n75nr_d843325d-498d-44b4-9900-75e7fcc48bf0       2d6h
webhook.conversionwebhook.00-of-01                                                                  tekton-pipelines-webhook-5976695855-c4gm6_cccfd8b7-1b3f-42ac-9796-58193374f20e      2d6h
webhook.defaultingwebhook.00-of-01                                                                  tekton-triggers-webhook-59df6955c8-n75nr_fb802917-af9d-4631-8a67-c83808a60026       2d6h
webhook.validationwebhook.00-of-01                                                                  tekton-triggers-webhook-59df6955c8-n75nr_8ec3c156-c9e9-4493-b6e2-78dbc0161d5c       2d6h
webhook.webhookcertificates.00-of-01                                                                tekton-triggers-webhook-59df6955c8-n75nr_c4daba5e-9512-4e49-9f41-4fecd2e0338a       2d6h

Pipeline and triggers use the same name webhook in their config:

• https://github.com/tektoncd/pipeline/blob/master/cmd/webhook/main.go#L235
• https://github.com/tektoncd/triggers/blob/58269c761dc5bb43e026fb67247b8e255da0f469/cmd/webhook/main.go#L121

However they use two different secret names:

$ k get secret -n tekton-pipelines | grep certs
triggers-webhook-certs                    Opaque                                3      2d6h
webhook-certs                             Opaque                                0      2d6h

The solution for this would be to use a namespaced name in the webhook config, webhook-pipeline for pipeline and webhook-trigger for triggers.

Also related work on knative side: knative/eventing#4530

Thanks @mattmoor for your help tracking this down!