Create Position around Management Cluster #8210
Conversation
k8s-ci-robot
added
the
cncf-cla: no
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: corentone The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
sig/multicluster
|
Welcome @corentone! |
|
Hi @corentone. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
the
size/M
| @@ -0,0 +1,46 @@ | |||
| # Management Cluster - SIG Multicluster Position Statement | |||
There was a problem hiding this comment.
I feel like the termManagement is a bit too generic. Could we do a poll first to gather some community feedback?
There was a problem hiding this comment.
This is a great idea, thank you! I'll ask in Slack for a list and try to figure out where/how to run said poll :)
| Kubernetes community to leverage a Kubernetes Cluster and the existing api-machinery | ||
| available. There has been a variety of examples of which we can quote ArgoCD, MultiKueue | ||
| or any of the Federation effort (Karmada, KubeAdmiral), all of them not-naming the "location" | ||
| where they run or not aligning on the name (Admin cluster, Hub Cluster, Manager Cluster...). |
There was a problem hiding this comment.
When you read Hub Cluster, you do think of the exact location of where it runs, right in the middle, especially if the term spoke comes up next.
There was a problem hiding this comment.
I didn't pick hub cluster because of two aspects for me:
1/ I was worried it'd sound weird when the "hub" cluster is part of the workload clusters themselves.
2/ given there can be multiple "hub clusters", hub may lose some meaning because it's not a singleton? multiple hubs is odd?
So I prioritized it's "function" over what it was? But those arguments are weak and someone suggested to poll the community, which is likely to be better than my arguments :)
| @@ -0,0 +1,46 @@ | |||
| # Management Cluster - SIG Multicluster Position Statement | |||
There was a problem hiding this comment.
Not all multi-cluster management designs have a monolith. I do not think this SIG should take a position that requires or recommends a monolith. There are multiple roles involved in multi-cluster management. I think that it may be more helpful to identify those roles. A system with a monolith can be described as having one cluster that plays all those roles.
For example, in https://github.com/kubestellar/kubestellar/ we identify distinct roles and allow flexibility in what plays which role. We start by identifying a concept that is less than a "cluster". We define a "space" to be the fragment of cluster behavior that is only concerned with generic API machinery stuff. A space can store and serve kube API objects and subjects them to the general-purpose controllers (the ones that apply to all kinds of API objects, not the controllers involved specifically with containerized workload (Pod, Service, ...) ). KubeStellar defines the following roles.
- an "inventory space" holds the inventory of managed clusters. That is, API objects that refer to or describe the managed clusters. In OCM these are
ManagedClusterobjects. - a "workload description space (WDS)" holds both (a) objects that describe the workload (both desired state from users and reported state to users) and (b) control objects that control the multi-cluster management behavior.
- a "workload execution cluster (WEC)" is a managed cluster, where the workload is run or executed.
- the current KubeStellar implementation uses OCM and has a "transport space" that holds the workload interface objects of OCM (e.g.,
ManifestWork). The current implementation of KubeStellar requires that one space play both "inventory space" and "transport space" roles, calling this joint role "inventory and transport space (ITS)". - the current implementation of KubeStellar uses https://github.com/kubestellar/kubeflex as a space manager. The KubeFlex API objects reside in, and the KubeFlex central controller runs in, a role called the "KubeFlex hosting cluster".
- KubeFlex considers itself to manage "ControlPlanes", and supports a few different types. One is "vcluster", which uses https://www.vcluster.com/ to create virtual slices of its hosting cluster. Another is "k8s", which is essentially hosted Kubernetes kube-apiserver, its storage, and general-purpose controllers (the essence of a "space"). KubeFlex can also have a ControlPlane that represents its hosting cluster. We are working on enabling KubeFlex to adopt an arbitrary existing cluster as a ControlPlane.
One configuration that KubeStellar supports is one real cluster playing the roles of WDS, ITS, and KubeFlex hosting cluster.
In OCM, where the workload description is wrapped and the workload execution cluster holds the unwrapped objects, could one cluster play both WDS and WEC roles?
There was a problem hiding this comment.
Thank you for those details. I agree with your statement of not being a monolith.
I'm only trying to define the nuance of what a management cluster is; I don't think I force everything to be in the same or a monolith. Maybe my wording is wrong? I could rename the title to be "Management Clusters" to make it clearer.... but I'm just trying to get a wide definition of what such a cluster is (by opposition to a workload cluster).
There was a problem hiding this comment.
I wonder what is Kubestellar recommended way to host WDS/ITS/Kubeflex?
| A (multicluster) management cluster is a Kubernetes cluster that acts as a | ||
| control-plane for other Kubernetes clusters (named Workload Clusters to differentiate | ||
| them). It MUST have visibility over the available clusters and MAY have administrative | ||
| privileges over them. It SHOULD not be part of workload clusters to provide a better |
There was a problem hiding this comment.
I am not convinced of this "SHOULD". I am not convinced that this sort of statement belongs in a definition.
There was a problem hiding this comment.
I wonder if you are not convinced that the "hub" cluster and a "spoke" cluster should not be the same or just "should" does not belong to a "definition"? I am curious about the reason if it's the former.
| them). It MUST have visibility over the available clusters and MAY have administrative | ||
| privileges over them. It SHOULD not be part of workload clusters to provide a better | ||
| security isolation, especially when it has any administrative privileges over them. | ||
| There MAY be multiple management clusters overseeing the same set of Workload Clusters |
There was a problem hiding this comment.
This statement supposes multiplicity only in the form of potentially competing equals; it omits the possibility of clusters fulfilling distinct roles.
There was a problem hiding this comment.
I tried to clarify in the next section that it could be multiple roles:
to allow for separation of functionality (security-enforcer management cluster vs
config-delivery management cluster)
is the wording not strong enough? I don't mean to close that door and thought the current wording was enough and not emitting a direction, just requiring that the admin oversees potential overlap between different management clusters. If there is no overlap, they are fine to co-exist as separate clusters.
| security isolation, especially when it has any administrative privileges over them. | ||
| There MAY be multiple management clusters overseeing the same set of Workload Clusters | ||
| and it is left to the administrator to guarantee that they don't compete in their | ||
| management tasks. There SHOULD be a single clusterset managed by a management cluster. |
There was a problem hiding this comment.
This begs the question of what is a "clusterset".
There was a problem hiding this comment.
https://multicluster.sigs.k8s.io/api-types/cluster-set/
I tried to tie it to the sig-mc definition (I need to link it, indeed!).
| config-delivery management cluster), to allow for migrations (from old management cluster to new | ||
| management cluster) and likely more. | ||
| * Management cluster also being part of the workload-running Fleet: We do recommend that the | ||
| management cluster(s) be isolated from the running Workload Fleet for security and management |
There was a problem hiding this comment.
This makes sense in general, but I am not convinced that there are no use cases for combining roles in one cluster.
There was a problem hiding this comment.
the first paragraph encourages different clusters for different roles actually.
This second paragraph is just about being part of the workload clusters or not.
Let me try to think of introducing the notion of "role" or something like that, as a subdivision of the broad Management Cluster.
There was a problem hiding this comment.
I wonder what's the definition of "workload"? I usually associate them with applications but not controllers so it's okey to me to run controllers in the "central" cluster that requires leader-election.
There was a problem hiding this comment.
it is definitely okay to run a controller in the management cluster. A controller doesn't have to be a "Workload" and be considered part of the "control-plane".
I think its the persona that matters. If it is a platform admin -owned controller, performing management tasks, I wouldn't consider it a workload. Workload to me is an application serving actual business-logic purpose.
All in all, running those management controllers is the reason why I want to define management clusters and not just a management API. (we had discussed internally giving simply an API with machinery... but then very quickly you want to bring a controller to act on this API and look where to run it)
k8s-ci-robot
added
cncf-cla: yes
| @@ -0,0 +1,46 @@ | |||
| # Management Cluster - SIG Multicluster Position Statement | |||
There was a problem hiding this comment.
Thank you for those details. I agree with your statement of not being a monolith.
I'm only trying to define the nuance of what a management cluster is; I don't think I force everything to be in the same or a monolith. Maybe my wording is wrong? I could rename the title to be "Management Clusters" to make it clearer.... but I'm just trying to get a wide definition of what such a cluster is (by opposition to a workload cluster).
| Kubernetes community to leverage a Kubernetes Cluster and the existing api-machinery | ||
| available. There has been a variety of examples of which we can quote ArgoCD, MultiKueue | ||
| or any of the Federation effort (Karmada, KubeAdmiral), all of them not-naming the "location" | ||
| where they run or not aligning on the name (Admin cluster, Hub Cluster, Manager Cluster...). |
There was a problem hiding this comment.
I didn't pick hub cluster because of two aspects for me:
1/ I was worried it'd sound weird when the "hub" cluster is part of the workload clusters themselves.
2/ given there can be multiple "hub clusters", hub may lose some meaning because it's not a singleton? multiple hubs is odd?
So I prioritized it's "function" over what it was? But those arguments are weak and someone suggested to poll the community, which is likely to be better than my arguments :)
| @@ -0,0 +1,46 @@ | |||
| # Management Cluster - SIG Multicluster Position Statement | |||
There was a problem hiding this comment.
This is a great idea, thank you! I'll ask in Slack for a list and try to figure out where/how to run said poll :)
| them). It MUST have visibility over the available clusters and MAY have administrative | ||
| privileges over them. It SHOULD not be part of workload clusters to provide a better | ||
| security isolation, especially when it has any administrative privileges over them. | ||
| There MAY be multiple management clusters overseeing the same set of Workload Clusters |
There was a problem hiding this comment.
I tried to clarify in the next section that it could be multiple roles:
to allow for separation of functionality (security-enforcer management cluster vs
config-delivery management cluster)
is the wording not strong enough? I don't mean to close that door and thought the current wording was enough and not emitting a direction, just requiring that the admin oversees potential overlap between different management clusters. If there is no overlap, they are fine to co-exist as separate clusters.
| security isolation, especially when it has any administrative privileges over them. | ||
| There MAY be multiple management clusters overseeing the same set of Workload Clusters | ||
| and it is left to the administrator to guarantee that they don't compete in their | ||
| management tasks. There SHOULD be a single clusterset managed by a management cluster. |
There was a problem hiding this comment.
https://multicluster.sigs.k8s.io/api-types/cluster-set/
I tried to tie it to the sig-mc definition (I need to link it, indeed!).
| config-delivery management cluster), to allow for migrations (from old management cluster to new | ||
| management cluster) and likely more. | ||
| * Management cluster also being part of the workload-running Fleet: We do recommend that the | ||
| management cluster(s) be isolated from the running Workload Fleet for security and management |
There was a problem hiding this comment.
the first paragraph encourages different clusters for different roles actually.
This second paragraph is just about being part of the workload clusters or not.
Let me try to think of introducing the notion of "role" or something like that, as a subdivision of the broad Management Cluster.
| @@ -0,0 +1,46 @@ | |||
| # Management Cluster - SIG Multicluster Position Statement | |||
There was a problem hiding this comment.
I wonder what is Kubestellar recommended way to host WDS/ITS/Kubeflex?
|
|
||
| A (multicluster) management cluster is a Kubernetes cluster that acts as a | ||
| control-plane for other Kubernetes clusters (named Workload Clusters to differentiate | ||
| them). It MUST have visibility over the available clusters and MAY have administrative |
There was a problem hiding this comment.
I wonder what does "visibility" mean here? There can be workload clusters that do not have public IP.
There was a problem hiding this comment.
Agreed, visibility is vague. I think it is somewhat the ability to query more data about the cluster. I'm thinking it would be the ability to "READ" via kubectl (thought it may not be everything). But it should be able to potentially see the Cluster object in GKE or other platforms (wherever the provisioning/lifecycle happens).
Maybe this is too vague, I originally had this in to mandate some kind of visibility on the cluster so that the controller could do something (if the controller has only access to the clusterprofile, I dont know what it can really do).
But we may be able to remove it and it wouldn't affect the definition much.
| A (multicluster) management cluster is a Kubernetes cluster that acts as a | ||
| control-plane for other Kubernetes clusters (named Workload Clusters to differentiate | ||
| them). It MUST have visibility over the available clusters and MAY have administrative | ||
| privileges over them. It SHOULD not be part of workload clusters to provide a better |
There was a problem hiding this comment.
I wonder if you are not convinced that the "hub" cluster and a "spoke" cluster should not be the same or just "should" does not belong to a "definition"? I am curious about the reason if it's the former.
| config-delivery management cluster), to allow for migrations (from old management cluster to new | ||
| management cluster) and likely more. | ||
| * Management cluster also being part of the workload-running Fleet: We do recommend that the | ||
| management cluster(s) be isolated from the running Workload Fleet for security and management |
There was a problem hiding this comment.
I wonder what's the definition of "workload"? I usually associate them with applications but not controllers so it's okey to me to run controllers in the "central" cluster that requires leader-election.
No description provided.