Dual-Stack support for Kubeadm

[Arvinderpal]

Is this a BUG REPORT or FEATURE REQUEST?

FEATURE REQUEST

In order to support the forthcoming dual-stack work, certain changes must be made to kubeadm so that it can create dual-stack clusters. The dual-stack KEP (kubernetes/enhancements#808) has a section on changes required in kubeadm.

Phase-1 Tasks:

• kubeadm feature-gate Adds kubeadm feature-gate for dual-stack (IPv6DualStack) kubernetes#80145
• Propagate kubeadm feature-gate to all k8s components Propagate kubeadm dual-stack feature-gate to all k8s components kubernetes#80531
• --cluster-cidr support for comma delimited list of IPs Dual-Stack Integration with Kubeadm kubernetes#79033
• e2e tests for --cluster-cidr/PodSubnet changes kubeadm e2e networking test for dual-stack podSubnet check. kubernetes#80947

Phase-2 Tasks:

• service-cluster-ip-range support for dual-stack. Kubeadm DualStack Support for List of Service IPs kubernetes#82473
• e2e tests for service-cluster-ip-range (these are in kind upstream)
• Add documentation on enabling dual-stack via kubadm
  Kubeadm/dual stack support in 1.21 website#26675

Additional Tasks:

• Option node-cidr-mask-size to kube-controller manager needs to support a list of CIDRs. This currently defaults to /24 (for both ipv4 and ipv6). Func calcNodeCidrSize must be updated to handle dual-stack. Add support for new dual-stack flags for kubernetes-controller-manager in kubeadm kubernetes#85609
• don't hardcode the ipv4 mask size to 24?
  Kubeadm incorrectly calculating the node CIDR in cases when the given podSubnet smaller than /24 #2327
  support variable sized pod-CIDRs for different-sized node (was: Allow --node-cidr-mask-size to change: make NodeIPAM controller support variable sized CIDRs) kubernetes#90922
  fixed in:
  kubernetes/kubernetes@8b52995#diff-d870edeb820f49abf2f52cd357a8a5396c7cd1a36e024d339146e512186c7297

GA (1.23):

• Add apiserver-advertise-address support for dual-stack
  looks like this is no longer a goal
• k/k PR that graduates the FG to GA:
  move IPv6DualStack feature to stable. kubernetes#104691
• move kubeadm's FG to GA + cleanup:
  cleanup: DualStack GA for kubeadm kubernetes#104770

post-GA (1.24)

• remove the kubeadm feature gate and extra code related to it
• cleanup docs pages such as:
  https://github.com/kubernetes/website/blob/082116588c191f52de2003deb71ce7a304cc357d/content/en/docs/setup/production-environment/tools/kubeadm/dual-stack-support.md

[neolit123]

thanks for logging the ticket @Arvinderpal
@aojea has been interested in implementing IPV6 support in kind, which uses kubeadm under the hood and adding testing signal for that.

@aojea please take a look and communicate with @Arvinderpal who wishes to add dual stack support in kubeadm.

/assign @aojea @Arvinderpal

[k8s-ci-robot]

@neolit123: GitHub didn't allow me to assign the following users: aojea.

Note that only kubernetes members and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

thanks for logging the ticket @Arvinderpal
@aojea has been interested in implementing IPV6 support in kind, which uses kubeadm under the hood and adding testing signal for that.

@aojea please take a look and communicate with @Arvinderpal who wishes to add dual stack support in kubeadm.

/assign @aojea @Arvinderpal

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[neolit123]

the IPV6 PRs merged in kind.
kubernetes-sigs/kind#636
also for test signal:
kubernetes/test-infra#13123
(should appear here https://k8s-testgrid.appspot.com/sig-testing-kind#kind%20(IPv6),%20master%20(dev))

[BenTheElder]

been keeping an eye on the dual stack work for exactly this @neolit123 👀

[aojea]

/cc

[pacoxu]

@neolit123 @aojea
Should we support multi node IPs in kubeadm init and join?

• auto-detect or with a flag --node-ips
• if dual-stack is true, detect both; if single-stack, prefer ipv4 or detect from pod-cidr first.

> --node-ip string IP address (or comma-separated dual-stack IP addresses) of the node. If unset, kubelet will use the node's default IPv4 address, if any, or its default IPv6 address if it has no IPv4 addresses. You can pass '::' to make it prefer the default IPv6 address rather than the default IPv4 address.

[aojea]

great catch, I think so, or your hostnetwork pods will be single-stack, that can be a problem ...

[neolit123]

well, given kubeadm already supports passing kubelet config / flags we should just document how users can do that (KubeletConfiguration == preferred).
i don't think we should expose this is a wrapper in the kubeadm config or kubeadm flags.

arguably the kubeadm flag --node-name should not exist for similar reasons, but it's already there.

[aojea]

documentation is fine, I didn't want to mean to create a new option

[pacoxu]

@neolit123 node-ips auto detect should be a kubelet feature in my mind.
Currently, we can add it to the docs for reminding users.

[aojea]

@neolit123 node-ips auto detect should be a kubelet feature in my mind.
Currently, we can add it to the docs for reminding users.

the user has to specify the IP addresses of each family, separated by comma, there is no autodetect for dual-stack

[pacoxu]

Per kubernetes/enhancements#563 (comment), it will wait to 1.23

[neolit123]

looks like the FG is moving to GA in 1.23:
kubernetes/kubernetes#104691

this mean we can remove the kubeadm FG + handling in kubeadm in 1.24 or later.

[pacoxu]

In 1.23 kubeadm, we may change the FG to hidden like below after kubernetes/kubernetes#104691 is merged.

IPv6DualStack:   {FeatureSpec: featuregate.FeatureSpec{Default: true, PreRelease: featuregate.GA}, HiddenInHelpText: true},

[khenidak]

Folks -- Do you want me to address gate removal in PR kubernetes/kubernetes#104691 - or do you want to take care of it separatly?

@neolit123 @pacoxu @aojea

[neolit123]

for 1.23 in kubeadm we should just change this line:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/features/features.go#L42

to be exactly this:

IPv6DualStack:   {FeatureSpec: featuregate.FeatureSpec{Default: true, LockToDefault: true, PreRelease: featuregate.GA}, HiddenInHelpText: true},

^ LockToDefault is true, i'm assuming this also has to happen for the core feature gate - i.e. when the FG goes GA we are going to lock it to true.
it also includes HiddenInHelpText: true as @pacoxu pointed out above (this is a kubeadm specific FG property).
you can do that in the kubernetes/kubernetes#104691 PR.

IMO, the cleanup of logic around the IPv6DualStack feature gate checks in kubeadm can be done when the kubeadm FG is removed. we can do that in 1.24 (one release after the FG goes GA).
not sure what the core maintainers prefer for the core logic, but making the kubeadm changes in 1.24 and separate PR seems like less noise.

seems like core generally prefers to remove the FG logic when a FG goes GA and not when the FG is removed - e.g.:
kubernetes/kubernetes#97543
(@pacoxu you did this change for LegacyNodeRoleBehavior)

[pacoxu]

seems like core generally prefers to remove the FG logic when a FG goes GA and not when the FG is removed

Yes. Either is ok(in 1.23 or 1.24).

@khenidak As Lubomir points out, you can just update the cmd/kubeadm/app/features/features.go#L42.

The removal of the FG logic in kubeadm can be done in your PR or a separate PR which is just a cleanup. Either is OK.

[khenidak]

Ok - I will update the feature file. But will not touch the rest. Thank you.

[neolit123]

we cleaned up the feature gate from kubeadm (1.24):
kubernetes/kubernetes#106648

thanks everyone for helping with this.