wip:Support multi control plane

[lingsamuel]

/kind feature

Changes:

• fix tests - hope finished
• fix missing InitialSetup in node add
• fix/implement node add --control-plane
• add flag --control-planes to config control planes number: minikube start --control-planes 2. --nodes will automatically enlarge if nodes < control-planes and outputs a warning.
• tag primary control plane: add field PrimaryControlPlane
• add IP and primary tag in status
• reset control plane node before stop it, make sure primary control plane can start without others control planes
• add some step output
• change KindNet.String() from CNI to KindNet
• status command: stacked control plane node kubeconfig should be irrelevant. kubeconfig still exist in stacked control plane node but configured at control-plane.minikube.internal:8443 to generate certificate keys.
• backward compatibility
  • start from a stopped old cluster config
  • node add, node stop, node start, stop a running old cluster
  • status warn user outdated config, provides flag --update-primary-control-plane-tag to update profile.
• multiple control planes integration tests

Fixes: #7461, part of #7538

Currently finished:

minikube start
minikube node add --control-plane # Support multi control plane
minikube stop # make sure stop && start works
minikube start
minikube node stop m02 # make sure node stop && node start works
minikube node start m02

Some output changes:

$ minikube status
minikube
type: Control Plane (Primary)
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured
timeToStop: Nonexistent
IP: 192.168.39.13

minikube-m02
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Irrelevant
timeToStop: Nonexistent
IP: 192.168.39.252

Outdated profile and status command output:

// ...
"Nodes": [
        {
            "Name": "",
            "IP": "192.168.39.164",
            "Port": 8443,
            "KubernetesVersion": "v1.19.3",
            "PrimaryControlPlane": false, // <- outdated config is false here
            "ControlPlane": true,
            "Worker": true
        }
    ],
// ...

$ minikube status
There is no primary control plane, set --update-primary-control-plane-tag=true to update profile.
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Irrelevant
timeToStop: Nonexistent
IP: 192.168.39.164

[minikube-bot]

Can one of the admins verify this patch?

[k8s-ci-robot]

Hi @lingsamuel. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: lingsamuel
To complete the pull request process, please assign priyawadhwa after the PR has been reviewed.
You can assign the PR to them by writing /assign @priyawadhwa in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[lingsamuel]

Rebased.

[lingsamuel]

Add a k8s version check, this may let tests pass.

[medyagh]

@lingsamuel thanks for updating this, I still need to verify if this PR could be accepted in minikube's vision without affection the code complexity. will do a discuss on next office hours if you like to attend on monday please we could discuss it

[lingsamuel]

I am afraid i can't participant.

[medyagh]

@lingsamuel thanks for this PR, at this release we dont have the bandwith to accept this PR but we eventually we can accept this.
the main concerns the use case for a local developement is hard to emulate, the end user would need to deploy a loadbalacner.

we decideded once our Flake Tests failure for multinode are fixed

[k8s-ci-robot]

@lingsamuel: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[medyagh]

@lingsamuel update: we discussed this in the triage meeting, we like to re-introduce this PR again after we deliver the minikube auto pause feature. #10326

for now I will close this PR and lets reopen a MEP for this feature for milestone v1.19.0
similar to https://github.com/kubernetes/minikube/tree/master/enhancements/proposed

I would like the MEP to discuss backward compatiblity, how it affects other minikube features such as pause, service, docker-env,....