Dynamic schema rfc #368
Dynamic schema rfc #368
Conversation
jakefhyde
force-pushed
the
dynamic-schema-rfc
branch
2 times, most recently
from
b0d9d0a
to
21692b9
Compare
jakefhyde
force-pushed
the
dynamic-schema-rfc
branch
3 times, most recently
from
98ec3e7
to
014357f
Compare
go.mod
Outdated
| k8s.io/dynamic-resource-allocation => k8s.io/dynamic-resource-allocation v0.29.3 | ||
| k8s.io/endpointslice => k8s.io/endpointslice v0.29.3 |
There was a problem hiding this comment.
Intellisense fails without doing this.
There was a problem hiding this comment.
Just as a heads up, we recently merged a PR for k8s 1.30 support - #412. If you could bump these to the version used in the PR I would appreciate it v0.30.1)
| // dynamicSchemaSpec field for a machine pool if the "provisioning.cattle.io/allow-dynamic-schema-drop" annotation is | ||
| // not present and true on the cluster. If the value of the annotation is true, no mutation is performed. | ||
| func (m *ProvisioningClusterMutator) handleDynamicSchemaDrop(request *admission.Request, cluster *v1.Cluster) (*admissionv1.AdmissionResponse, error) { | ||
| if cluster.Name == "local" || cluster.Spec.RKEConfig == nil { |
There was a problem hiding this comment.
we should make sure the harvester team isn't somehow relying on dynamic schema for the local cluster in rancherd
There was a problem hiding this comment.
i don't think we are leveraging dynamic schema on local cluster for anything during the harvester bootstrap / lifecycle management. @bk201 @Vicente-Cheng
There was a problem hiding this comment.
Sounds good then, as long as harvester isn't worried about the dynamic schema changing with new flags and causing reprovisioning for the local cluster (not sure if that's something that's been encountered previously, or if its even possible for rancherd to bootstrap the local cluster using harvester).
There was a problem hiding this comment.
@FrankYang0529 Please help take a look too.
There was a problem hiding this comment.
We don't use MachinePools field in RKEConfig, but we use RotateCertificates. Not sure whether it's related? Thanks.
There was a problem hiding this comment.
@FrankYang0529 Yeah, not realted to any of the day 2 ops, only when new fields are added to a driver.
jakefhyde
force-pushed
the
dynamic-schema-rfc
branch
from
014357f
to
0cdb615
Compare
jakefhyde
force-pushed
the
dynamic-schema-rfc
branch
from
14d96c4
to
887fd58
Compare
pkg/resources/provisioning.cattle.io/v1/cluster/mutator_test.go
Outdated
Show resolved
Hide resolved
pkg/resources/provisioning.cattle.io/v1/cluster/mutator_test.go
Outdated
Show resolved
Hide resolved
pkg/resources/provisioning.cattle.io/v1/cluster/mutator_test.go
Outdated
Show resolved
Hide resolved
Issue: rancher/rancher#44618
Problem
An issue was identified (rancher/rancher#40609), where updating the node driver and changing the flags would result in CAPI detecting the supplied fields do not match the existing fields on the machine infrastructure custom resource. The fix for this issue was to persist the dynamicSchema onto the cluster object under the machinePools section, and stripping fields not present at the time of initial provisioning from the related CR, as the alternative of properly versioning CRDs for each node driver was not and continues to not be a viable solution. While this change initially fixed the issue of unintended cluster reprovisioning on node driver update, other cluster update solutions such as terraform and kubectl are still susceptible to this issue as an update unintentionally removing the dynamicSchemaSpec fields will still result in reprovisioning if new fields (whether used or not) are added to a node driver. Since the dynamicSchemaSpec is added by a controller within Rancher, a cluster create request followed by updating the node driver and a subsequent update request (identical to the cluster create request) will cause reprovisioning, due to the dynamicSchemaSpec being unpopulated within the request.
Solution
Add an annotation which the webhook will use to determine whether the removal of the dynamicSchemaSpec is intentional. If not present or ”false”, the dynamicSchemaSpec will be reinserted into the cluster object as-is, preventing an unintentional cluster-wide machine rollout. If true, the fields will stay removed from the request, and allow provisioning with the previous behavior, allowing users to opt-in for a potential rollout if the node driver was indeed changed.
CheckList
There isn't currently an existing markdown file for provisioning clusters. I'm happy to create one in a follow up PR if necessary.