Support out-of-process and out-of-tree cloud providers

[errordeveloper]

Feature Description:
Support out-of-tree and out-of-process cloud providers, a.k.a pluggable cloud providers.

Feature Progress:
In order to complete this feature, cloud provider dependencies need to be moved out the the following Kubernetes binaries, then docs and tests need to be added. The Links to the right hand side of the binary denote the PRs that lead to the completion of the sub-feature

1. Kube-controller-manager -

• start breaking up controller manager into two pieces kubernetes#34273,
• No need to ScrubDNS anymore kubernetes#36785,
• make private method private kubernetes#39394,
• add "external" cloud provider kubernetes#41856,
• Use ProviderID to address nodes in the cloudprovider kubernetes#42604,
• move ProvideID indexed methods to right location kubernetes#43777

2. Kubelet

• cloud initialize node in external cloud controller kubernetes#44258

3. Docs

• add docs about cloud-controller-manager website#2900

4. Tests

e2e Tests - Incomplete

The cloud-specific functionality of the above features needs to be moved into a new binary called cloud-controller-manager that support a plugin architecture.

Primary Contact: @wlan0

Responsible SIG: @k8s-mirror-cluster-lifecycle-feature-re

Design Proposal Link: kubernetes/community#128

Reviewers:
@luxas
@roberthbailey
@thockin

Approver:
@thockin

Feature Target:
Alpha: 1.7
Beta: 1.8
Stable: 1.10

---

Here's an updated status report for this feature, please let me know if anything needs clarification:

Beta (starting v1.11)

• The common interface used by cloud providers has been well tested and support will not be dropped, though implementation details may change. Any methods that are deprecated should follow the Kubernetes Deprecation Policy.
• The cloud controller manager has been tested by various cloud providers and is considered safe to use for out-of-tree providers. Features to be deprecated that are part of the cloud controller manager (controllers, component flags, etc) will follow the Kubernetes Deprecation Policy.
• The cloud controller manager does not run in any cluster by default. It must be explicitly turned on and added like any other control plane component. Instructions for setup may slightly vary per cloud provider. More details here.

Reasoning for Graduation

There were a few things on our TODO list that we wanted to get done before graduating to beta such as collecting E2E tests from all providers & improving out-of-tree storage. However, many of these initiatives require collaboration from external parties that was delaying progress on this effort. In addition, there was uncertainty since we do not develop some of the components we rely on, a good example is whether CSI would be able to meet demands for out-of-tree storage that was on par with in-tree storage support. Though in hindsight we have more confidence in CSI, prior to its beta release it was unclear if it would meet our requirements. With this context in mind, we had decided to graduate to beta because:

• blocking out-of-tree cloud providers from going beta meant that less in-tree providers will adopt this feature.
• some goals (like E2E tests from cloud providers) requires a significant amount of collaboration and may unnecessarily block progress for many releases.
• features that are lacking from the cloud controller manager (mainly storage) would be handled by future projects from other SIGs (e.g. CSI by SIG Storage).

Goals for GA (targetted for v1.13/v1.14)

• Frequently collect E2E tests results from all in-tree & out-of-tree cloud providers SIG Cloud Provider KEP: Reporting Conformance Test Results to Testgrid community#2224
• Cloud Provider Documentation includes:
  • “Getting Started” documentation - outlines the necessary steps required to stand up a Kubernetes cluster.
  • Documentation outlining all cloud provider features such as LoadBalancers, Volumes, etc. There should be docs providing a high-level overview and docs that dig into sufficient details on how each feature works under the hood.
  • Docs should also be centralized in an automated fashion where documentation from all cloud providers are placed into a central location (ideally https://kubernetes.io/docs/home/).
• A well-documented plan exists for how to migrate a cluster from using in-tree cloud provider to out-of-tree cloud provider, this only applies to AWS, Azure, GCP, OpenStack, and VMWare.
• All current cloud providers have implemented an out-of-tree solution, deprecation of in-tree code is preferred but not a requirement.

[colemickens]

Benefits:

• Easier configuration for providers like Azure that require a "cloud config" flag on kubelet/kcm. This file could instead by made a Secret (or ConfigMap + Secret). Makes bootstrapping easier and would eliminate the need for kubeadm to have special functionality for handling the cloudprovider flags.
• Selectively enablement. Some people want to run their own overlay network, but still want auto-provisioned L4 load balancers. There's no way to do that today.
• Moves more things out of core Kubernetes repo/project, and enables faster turn-around for shipping new cloudproviders or iterating/testing changes.

Just a note, kubelet uses cloudprovider too, in addition to KCM.

[errordeveloper]

cc @kubernetes/sig-cluster-lifecycle @kubernetes/sig-network @kubernetes/sig-storage @kubernetes/sig-aws @kubernetes/sig-openstack

[thockin]

I endorse this idea in general. I think the built-in cloud provider logic has served its purpose and its time to modularize. I think there are a number of facets to this that we have to work out including but not limited to:

• CloudProvider and all the APIs therein
• Volume drivers and provisioner support
• Cluster turnup support

I think it would be worthwhile to start building a doc that details these and explores options for ejecting each one. I don't think there's anything here that hasn't been considered at SOME point. Once we get that written down, we can craft a roadmap...

[idvoretskyi]

Sounds useful. @errordeveloper, do we have any mailing lists or GitHub discussions with this question what we can refer to?

[errordeveloper]

@idvoretskyi not yet, this is probably very much on the radar of @kubernetes/sig-cluster-lifecycle.

[thockin]

I don't know that anyone is working on speccing this. It touches on a few
SIGs, but it is not exactly any of them.

On Tue, Sep 20, 2016 at 5:35 AM, Ilya Dmitrichenko <notifications@github.com

wrote:

@idvoretskyi https://github.com/idvoretskyi not yet, this is probably
very much on the radar of @kubernetes/sig-cluster-lifecycle
https://github.com/orgs/kubernetes/teams/sig-cluster-lifecycle.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#88 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVJJcJXb3gnueygHKf_uNVH7GkQ2nks5qr9MLgaJpZM4J6QkU
.

[errordeveloper]

@thockin you are right. May be we should form sig-cloud?

[thockin]

so. many. sigs. I don't think we need a SIG for this. I doubt if it is
going to garner much resistance. There are just a lot of details to hammer
out. Being on the radar for lifecycle is fine. The hardest part here is
balancing the desire for modularity with the need for simplicity. That's
what I want to see explored :)

On Wed, Sep 21, 2016 at 12:07 AM, Ilya Dmitrichenko <
notifications@github.com> wrote:

@thockin https://github.com/thockin you are right. May be we should
form sig-cloud?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#88 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVA1M1spBOe4hF4Wsgy-YQsvppxUeks5qsNeygaJpZM4J6QkU
.

[idvoretskyi]

@errordeveloper no need in yet another SIG (SIG-Cloud sounds like an abstract and umbrella one). I agree with @thockin - the primary SIG has to be @kubernetes/sig-cluster-lifecycle; while on behalf of @kubernetes/sig-openstack I'm going to track this item.

Hope other cloud-SIG's will be involved in the process as well.

[errordeveloper]

@justinsb and I have discussed this on Slack, and looks like we may be able to get closer to getting similar user-facing value by exposing flags via component config. It also turns out --configure-cloud-routes is already there. It doesn't look like this should involve moving code as such.

[bboreham]

I think there is additional value to moving code out to add-ons: it will enable further cloud providers to be added without enlarging the core of Kubernetes.

Example: kubernetes/kubernetes#32419

[errordeveloper]

Ah, but it also looks like someone is working on this:
kubernetes/kubernetes#32419 (comment).

On Wed, 28 Sep 2016, 18:19 Bryan Boreham, notifications@github.com wrote:

I think there is additional value to moving code out to add-ons: it will
enable further cloud providers to be added without enlarging the core of
Kubernetes.

Example: kubernetes/kubernetes#32419
kubernetes/kubernetes#32419

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#88 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPWS7hzMWW-U6R-KrehxlYilUUXAIqnks5quqGkgaJpZM4J6QkU
.

[thockin]

That does not read as someone working on it, to me. This is a big problem
with a lot of facets, and it needs a capital-O Owner.

On Wed, Sep 28, 2016 at 10:27 AM, Ilya Dmitrichenko <
notifications@github.com> wrote:

Ah, but it also looks like someone is working on this:
kubernetes/kubernetes#32419 (comment)
.

On Wed, 28 Sep 2016, 18:19 Bryan Boreham, notifications@github.com
wrote:

I think there is additional value to moving code out to add-ons: it will
enable further cloud providers to be added without enlarging the core of
Kubernetes.

Example: kubernetes/kubernetes#32419
kubernetes/kubernetes#32419

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#88 (comment)
,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAPWS7hzMWW-U6R-
KrehxlYilUUXAIqnks5quqGkgaJpZM4J6QkU>
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#88 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVHiMNL34Eyg1T0vaRtIvQ065ssuTks5quqOKgaJpZM4J6QkU
.