Skip to content

Commit 612e7c6

Browse files
lentzi90lentzi90
committed
Multi-tenancy proposal
This adds a proposal for how to implement the CAPI multi-tenancy contract for CAPM3. Signed-off-by: Lennart Jern <lennart.jern@est.tech>
1 parent b657702 commit 612e7c6

File tree

1 file changed

+221
-0
lines changed

1 file changed

+221
-0
lines changed

‎design/cluster-api-provider-metal3/multi-tenancy_contract.md‎

Lines changed: 221 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,221 @@
1+
<!--
2+
This work is licensed under a Creative Commons Attribution 3.0
3+
Unported License.
4+
5+
http://creativecommons.org/licenses/by/3.0/legalcode
6+
-->
7+
8+
# CAPM3 Multi-Tenancy Contract
9+
10+
## Status
11+
12+
implementable
13+
14+
## Summary
15+
16+
Implement the CAPI multi-tenancy contract for the Metal3 infrastructure provider
17+
by adding a new field to the Metal3Cluster and Metal3MachineTemplate resource.
18+
The new field will be a reference to a secret that contains the credentials for
19+
accessing the BareMetalHosts.
20+
21+
## Motivation
22+
23+
It is currently hard to use the Metal3 infrastructure provider in a multi-tenant
24+
environment. Some options are available (separate namespaces, separate
25+
CAPM3/BMO/Ironic instances), but they all come with drawbacks. Considering that
26+
there is a well defined CAPI contract for multi-tenancy, we should implement it.
27+
28+
### Goals
29+
30+
- Implement the CAPI multi-tenancy contract for the Metal3 infrastructure
31+
provider
32+
- Allow CAPM3 to work with BareMetalHosts in another cluster or namespace than
33+
where the Metal3Machine is.
34+
35+
### Non-Goals
36+
37+
- Removing or changing how CAPM3 is associating Metal3Machines with the
38+
BareMetalHosts.
39+
- Solving issues related to multiple CAPM3 instances working with the same
40+
BareMetalHosts. This should be handled separately, e.g. by introducing a
41+
[claim resource](https://github.com/metal3-io/metal3-docs/pull/408).
42+
- Limiting the scope of the credentials that CAPM3 needs to access the
43+
BareMetalHosts. This should be handled separately, e.g. by introducing a
44+
[claim resource](https://github.com/metal3-io/metal3-docs/pull/408).
45+
- To propagate back the relevant information to the Metal3Machine from the
46+
BareMetalHost.
47+
48+
## Proposal
49+
50+
Add a new field to the Metal3Cluster and Metal3MachineTemplate resource that
51+
will be a reference to a secret that contains the credentials for accessing the
52+
BareMetalHosts. The new field would also propagate from the
53+
Metal3MachineTemplate to the Metal3Machine. CAPM3 should use the credentials for
54+
associating Metal3Machines with the BareMetalHosts.
55+
56+
### User Stories
57+
58+
#### Story 1
59+
60+
As an administrator of bare-metal resources, I want to be able to create
61+
separate pools of BareMetalHosts for different users so that they cannot
62+
interfere with each other.
63+
64+
#### Story 2
65+
66+
As an administrator of bare-metal resources, I want to be able to create common
67+
pools of BareMetalHosts that two or more users can collaborate on.
68+
69+
#### Story 3
70+
71+
As a user of CAPM3, I want to make use of the BareMetalHosts that another team
72+
has created and made available to me, without necessarily sharing the same
73+
cluster.
74+
75+
#### Story 4
76+
77+
As a one-man-show, I want to continue using the current single-tenant setup with
78+
in-cluster credentials for accessing the BareMetalHosts. This is especially
79+
important for bootstrapping a self-hosted management cluster.
80+
81+
#### Story 5
82+
83+
As an advanced user of CAPM3, I want to be able to build clusters spanning over
84+
multiple sets of BareMetalHosts, each with their own credentials, for improved
85+
resilience.
86+
87+
### Implementation Details
88+
89+
The new Metal3Cluster field will be `.spec.identityRef` with sub-fields `name`
90+
and `context` (optional). The `name` should point to a secret in the same
91+
namespace as the Metal3Cluster, containing a kubeconfig file. The optional
92+
`context` should point to a context in the kubeconfig file. If the `context` is
93+
not specified, the current context will be used.
94+
95+
For the Metal3MachineTemplate, the new field will be
96+
`.spec.template.spec.identityRef` with the same sub-fields as the Metal3Cluster.
97+
This will propagate to the Metal3Machine's `.spec.identityRef`.
98+
99+
If the `identityRef` is not specified, CAPM3 should use in-cluster credentials
100+
for accessing the BareMetalHosts (current behavior). If the `identityRef` is
101+
specified on the Metal3Cluster, but not on the Metal3MachineTemplate, CAPM3
102+
should use the credentials specified in the Metal3Cluster for all
103+
Metal3Machines.
104+
105+
CAPM3 should make use of the kubeconfig file to access the BareMetalHosts
106+
related to the Metal3Cluster. The kubeconfig file can point to a namespace in
107+
the same cluster or to a remote cluster. Secrets for metadata, networkdata and
108+
userdata must also be created next to the BareMetalHosts.
109+
110+
Example:
111+
112+
```yaml
113+
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
114+
kind: Metal3Cluster
115+
metadata:
116+
name: cluster-1
117+
spec:
118+
identityRef:
119+
name: cluster-1-kubeconfig
120+
context: cluster-1-context
121+
---
122+
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
123+
kind: Metal3MachineTemplate
124+
metadata:
125+
name: cluster-1-controlplane-template
126+
spec:
127+
template:
128+
spec:
129+
identityRef:
130+
name: control-plane-bmh-kubeconfig
131+
context: default
132+
```
133+
134+
The status of the Metal3Machine will not be changed as part of this proposal.
135+
Users wishing to see details about the BareMetalHost, will have to access it
136+
directly, just as before this proposal. Relevant errors or conditions should
137+
always be propagated up anyway for visibility, this proposal does not change
138+
that.
139+
140+
The status of the Metal3Cluster does not need to change. However, the Ready
141+
condition should be set only when CAPM3 has verified that the identityRef
142+
credentials are valid. Invalid credentials should result in an error on the
143+
Metal3Cluster.
144+
145+
#### Meta data, user data and network data
146+
147+
Metal3DataTemplates, Metal3DataClaim and Metal3Data objects would not change
148+
with the implementation of this proposal, with the exception that the resulting
149+
secrets would be located next to the BareMetalHosts. The custom resources would
150+
be with the Metal3Machines, but the secrets must be with the BareMetalHosts so
151+
that the Bare Metal Operator can make use of them.
152+
153+
Note that the `.status.metaData` and `.status.userData` of a Metal3Machine would
154+
be located in the same namespace and cluster as the BareMetalHost. The
155+
`.status.renderedData` on the other hand would be located in the same namespace
156+
as the Metal3Machine.
157+
158+
### Risks & Mitigations
159+
160+
Considering that BareMetalHosts could be in a different cluster entirely, the
161+
move operation will be more complex. BareMetalHosts in external clusters should
162+
not be moved at all. BareMetalHosts in the same namespace and cluster should be
163+
handled as usual. An open question is how to handle BareMetalHosts in a
164+
different namespace.
165+
166+
In practice this is not really a Metal3 concern since it is a CAPI feature that
167+
we are using. CAPI will only move the CAPI/CAPM3 resources and other resources
168+
that are labeled to be moved. We will need to document it properly though so
169+
that users knows what to expect and in which cases the move operation will work.
170+
171+
### Work Items
172+
173+
- Add the new `identityRef` field to the Metal3Cluster, Metal3MachineTemplate,
174+
and Metal3Machine resource and update CAPM3 to make use of it.
175+
- Add E2E test (or modify existing test to use the new field).
176+
- Update documentation.
177+
178+
## Dependencies
179+
180+
None
181+
182+
## Test Plan
183+
184+
- Unit tests
185+
- At least one E2E test should use the new field and at least one E2E test
186+
should use in-cluster credentials.
187+
188+
## Upgrade / Downgrade Strategy
189+
190+
It will not be possible to downgrade to a version without support for the new
191+
field when making use of the new field.
192+
193+
## Drawbacks
194+
195+
None
196+
197+
## Alternatives
198+
199+
- Handle multi-tenancy through namespace isolation with a single cluster-wide
200+
CAPM3 instance. Users cannot share a BMH pool without working in the same
201+
namespace. The cluster-wide CAPM3 instance would have access to all the
202+
BareMetalHosts.
203+
- Handle multi-tenancy through namespace isolation with per namespace CAPM3
204+
instances. This is impractical since all CAPM3 instances would need to be of
205+
the same version (since CRDs are global). The same issue goes for per
206+
namespace BMO instances. Users cannot share a BMH pool without working in the
207+
same namespace.
208+
209+
None of these alternatives provide any way of separating the BareMetalHosts from
210+
the Metal3Machines.
211+
212+
- The `identityRef` field could allow referencing different kinds of credentials
213+
also. For example, a service account token. This may be simpler for users
214+
sharing a cluster, but it would not be possible to reference other clusters. A
215+
kubeconfig file covers all the use cases with a slight increase in complexity
216+
for same-cluster access and is therefore preferred.
217+
218+
## References
219+
220+
1. [CAPI multi-tenancy
221+
contract](https://cluster-api.sigs.k8s.io/developer/architecture/controllers/multi-tenancy#contract)

0 commit comments

Comments
 (0)