You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Improve CRD maintenance options and fix permissions #839
A single setting is too limited and currently breaks operator installations in more ways than it needs to. I therefor propose to implement the following changes.
Note
All option names are up for debate. The ones below are just the ones I came up with while writing the proposal.
There should be an option to toggle CRD creation by the operator. This could look like:
maintenance:
customResourceDefinitions:
create: true # (default) or false
This would allow users to opt-out of CRD creation. The create permission on apiextensions.k8s.io/customresourcedefinitions should be appropriately gated.
There should be an option to toggle the conversion webhook. This could look like:
maintenance:
customResourceDefinitions:
conversion: true # (default) or false
This would allow users to opt-out of creating and running the conversion webhook as part of the operator. It should be noted that opting out of running the conversion webhook requires everyone and everything to use the latest available CRD version going forward as no automatic conversion between older and never CRD versions is performed.
Remove the existing option, because in combination with two new options this would result in the following (confusing) combination matrix:
\
maintain
create
conversion
Notes
1
false
false
false
No CRDs are created by the operator, different mechanism needed. The CRD also isn't maintained, which it also doesn't need to, because the conversion is also turned off*.
2
true
false
false
No CRDs are created by the operator, different mechanism needed. It is requested to maintain the CRD, but it doesn't need to, because the conversion is turned off*.
3
false
true
false
The CRDs are created. The CRD isn't maintained, which it also doesn't need to, because the conversion is also turned off*.
4
false
false
true
No CRDs are created by the operator, different mechanism needed. The CRD also isn't maintained, but the conversion webhook runs. It is however not used, because the conversion webhook configuration is not injected*.
5
true
true
false
The CRDs are created. It is requested to maintain the CRD, but it doesn't need to, because the conversion is turned off*.
6
false
true
true
The CRDs are created. The CRD isn't maintained, but the conversion webhook runs. It is however not used, because the conversion webhook configuration is not injected*.
7
true
false
true
No CRDs are created by the operator, different mechanism needed. The CRD is maintained and the conversion webhook runs. I guess this is useful for deployments where the CRD should be created by a different deployment mechanism, like ArgoCD.
8
true
true
true
The complete deal. Everything works as expected.
This can and should be reduced to the following combination matrix:
\
create
conversion
Notes
1
false
false
No CRDs are created by the operator, different mechanism needed. Also, no CRDs are maintained and no conversion webhook is running*.
2
true
false
The CRDs are created. The conversion webhook is not running*.
3
false
true
No CRDs are created by the operator, different mechanism needed. Once CRDs are established, they can be maintained and converted via the conversion webhook.
4
true
true
The complete deal. Everything works as expected.
I would argue this is way easier to grasp and also what users probably expect when using these options.
* Only the latest CRD version can be used.
There should be an option to toggle deploying default custom resource objects. This could look like:
maintenance:
customResourceDefinitions:
createDefault: true # (default) or false
In addition to the option changes above, I would also like to overhaul how permissions are gated:
For operators which deploy default custom resource objects for their own CRDs (like listener and secret), the permissions to create and patch these objects should only be granted when createDefault is set to true.
The permissions to list and watchapiextensions.k8s.io/customresourcedefinitions is always needed as long as we have the crd_established startup condition.
The permissions to createapiextensions.k8s.io/customresourcedefinitions is only needed when create is set to true. The patch permission is only needed when the CRD needs to be maintained. This is the case when conversion is set to true.
Just adding the new options leads slightly weird naming. I would suggest the following names:
Nice!
I would only suggest that createDefaults is more descriptive, like createDefaultResources. convert could become enableConversions (if we want it to look similar to others).
I agree that the "smaller" matrix with two options makes more sense
When this is implemented the caBundle writing needs to work independently from the createDefinitions setting, otherwise row 3 (someone else owns the CRD, operator keeps conversion alive) still does not work which to me seems like one of the biggest failures so far
When this is implemented the caBundle writing needs to work independently from the X setting
I agree with Nick that createDefaultResources and enableConversions sounds better
[...] The patch permission is only needed when the CRD needs to be maintained. This is the case when conversion is set to true.
Claude has convinced me that this is false. We need patch also for createDefinitions and enableConversions and suggests this instead:
For operators which deploy default custom resource objects for their own CRDs (like listener and secret), the permissions on these objects should only be granted when createDefaultResources is set to true. create is enough, since these are only ever created if missing and never patched.
The permissions to list and watchapiextensions.k8s.io/customresourcedefinitions are always needed as long as we have the crd_established startup condition.
The permission to createapiextensions.k8s.io/customresourcedefinitions is only needed when createDefinitions is set to true.
The patch permission is needed when createDefinitionsorenableConversions is set to true, because both write to the CRD through the same server-side apply, and that apply is checked against patch for every call after the CRD exists.
Expected effort
7-10 days. If it takes longer let me know, or if you think it'll take longer
I had this in mind for quite some time, but was re-triggered by @NickLarsenNZ's work on #798.
Currently, there is a single CRD maintenance option available in all operators:
A single setting is too limited and currently breaks operator installations in more ways than it needs to. I therefor propose to implement the following changes.
Note
All option names are up for debate. The ones below are just the ones I came up with while writing the proposal.
There should be an option to toggle CRD creation by the operator. This could look like:
This would allow users to opt-out of CRD creation. The
createpermission onapiextensions.k8s.io/customresourcedefinitionsshould be appropriately gated.There should be an option to toggle the conversion webhook. This could look like:
This would allow users to opt-out of creating and running the conversion webhook as part of the operator. It should be noted that opting out of running the conversion webhook requires everyone and everything to use the latest available CRD version going forward as no automatic conversion between older and never CRD versions is performed.
Remove the existing option, because in combination with two new options this would result in the following (confusing) combination matrix:
This can and should be reduced to the following combination matrix:
I would argue this is way easier to grasp and also what users probably expect when using these options.
* Only the latest CRD version can be used.
There should be an option to toggle deploying default custom resource objects. This could look like:
In addition to the option changes above, I would also like to overhaul how permissions are gated:
createandpatchthese objects should only be granted whencreateDefaultis set totrue.listandwatchapiextensions.k8s.io/customresourcedefinitionsis always needed as long as we have thecrd_establishedstartup condition.createapiextensions.k8s.io/customresourcedefinitionsis only needed whencreateis set totrue. Thepatchpermission is only needed when the CRD needs to be maintained. This is the case whenconversionis set totrue.Just adding the new options leads slightly weird naming. I would suggest the following names: