Skip to content

Commit beff1e1

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request kubernetes-sigs#2839 from fabriziopandini/book-developing-e2e-test
📖 developing e2e tests
2 parents 15300a4 + 8476213 commit beff1e1

File tree

3 files changed

+230
-37
lines changed

3 files changed

+230
-37
lines changed

‎docs/book/src/SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525
- [Repository Layout](./developer/repository-layout.md)
2626
- [Rapid iterative development with Tilt](./developer/tilt.md)
2727
- [Testing](./developer/testing.md)
28+
- [Developing E2E tests](./developer/e2e.md)
2829
- [Controllers](./developer/architecture/controllers.md)
2930
- [Bootstrap](./developer/architecture/controllers/bootstrap.md)
3031
- [Cluster](./developer/architecture/controllers/cluster.md)

‎docs/book/src/developer/e2e.md‎

Lines changed: 228 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,228 @@
1+
# Developing E2E tests
2+
3+
E2E tests are meant to verify the proper functioning of a Cluster API management
4+
cluster in an environment that resemble a real production environment.
5+
6+
Following guidelines should be followed when developing E2E tests:
7+
8+
- Use the [Cluster API test framework].
9+
- Define test spec reflecting real user workflow, e.g. [Cluster API quick start].
10+
- Unless you are testing provider specific features, ensure your test can run with
11+
different infrastructure providers (see [Writing Portable Tests](#writing-portable-e2e-tests)).
12+
13+
The [Cluster API test framework] provides you a set of helpers method for getting your test in place
14+
quickly; the [test E2E package] provide examples of how this can be achieved and reusable
15+
test specs for the most common Cluster API use cases.
16+
17+
## Prerequisites
18+
19+
Each E2E test requires a set of artifacts to be available:
20+
21+
- Binaries & docker images for Kubernetes, CNI, CRI & CSI
22+
- Manifests & docker images for the Cluster API core components
23+
- Manifests & docker images for the Cluster API infrastructure provider; in most cases
24+
also machine images are required (AMI, OVA etc.)
25+
- Credentials for the target infrastructure provider
26+
- Other support tools (e.g. kustomize, gsutil etc.)
27+
28+
The Cluster API test framework provides support for building and retrieving the manifest
29+
files for Cluster API core components and for the Cluster API infrastructure provider
30+
(see [Setup](#setup))
31+
32+
For the remaining tasks you can find examples of
33+
how this can be implemented e.g. in [CAPA E2E tests] and [CAPG E2E tests].
34+
35+
## Setup
36+
37+
In order to run E2E tests it is required to create a Kubernetes cluster with a
38+
complete set of Cluster API providers installed. Setting up those elements is
39+
usually implemented in a `BeforeSuite` function, and it consists of two steps:
40+
41+
- Defining an E2E config file
42+
- Creating the management cluster and installing providers
43+
44+
### Defining an E2E config file
45+
46+
The [E2E config file] provides a convenient and flexible way to define common tasks for
47+
setting up a management cluster.
48+
49+
Using the config file it is possible to:
50+
51+
- Define the list of providers to be installed in the management cluster. Most notably,
52+
for each provider it is possible to define:
53+
- One or more versions of the providers manifest (built from the sources, or pulled from a
54+
remote location).
55+
- A list of additional files to be added to the provider repository, to be used e.g.
56+
to provide `cluster-templates.yaml` files.
57+
- Define the list of variables to be used when doing `clusterctl init` or
58+
`clusterctl config cluster`.
59+
- Define a list of intervals to be used in the test specs for defining timeouts for the
60+
wait and `Eventually` methods.
61+
- Define the list of images to be loaded in the management cluster (this is specif of
62+
management cluster based on kind).
63+
64+
An [example E2E config file] can be found here.
65+
66+
<aside class="note">
67+
68+
<h1>Deprecated E2E config file format</h1>
69+
70+
The [Cluster API test framework] includes also a [deprecated E2E config file] implementation,
71+
that was used before the introduction of clusterctl. This might be removed in future releases
72+
of the test framework.
73+
74+
</aside>
75+
76+
### Creating the management cluster and installing providers
77+
78+
In order to run Cluster API E2E tests, you need a Kubernetes cluster; the [NewKindClusterProvider] gives you a
79+
type that can be used to create a local kind cluster and pre-load images into it, but also existing clusters can
80+
be used if available.
81+
82+
Once you have a Kubernetes cluster, the [InitManagementClusterAndWatchControllerLogs method] provides a convenient
83+
way for installing providers.
84+
85+
This method:
86+
- Runs `clusterctl init` using the above local repository.
87+
- Waits for the providers controllers to be running.
88+
- Creates log watchers for all the providers
89+
90+
<aside class="note">
91+
92+
<h1>Deprecated InitManagementCluster method</h1>
93+
94+
The [Cluster API test framework] includes also a [deprecated InitManagementCluster method] implementation,
95+
that was used before the introduction of clusterctl. This might be removed in future releases
96+
of the test framework.
97+
98+
</aside>
99+
100+
## Writing test specs
101+
102+
A typical test spec is a sequence of:
103+
104+
- Creating a namespace to host in isolation all the test objects
105+
- Creating objects in the management cluster, wait for the corresponding infrastructure to be provisioned.
106+
- Exec operations like e.g. changing the Kubernetes version or `clusterctl move`, wait for the action to complete.
107+
- Delete objects in the management cluster, wait for the corresponding infrastructure to be terminated.
108+
109+
### Creating Namespaces
110+
111+
The [CreateNamespaceAndWatchEvents method] provides a convenient way to create a namespace and setup
112+
watches for capturing namespaces events
113+
114+
### Creating objects
115+
116+
There are two possible approaches for creating objects in the management cluster:
117+
118+
- Create object by object: create the `Cluster` object, then `AwsCluster`, `Machines`, `AwsMachines` etc.
119+
- Apply a `cluster-templates.yaml` file thus creating all the objects this file contains.
120+
121+
The first approaches leverage on the [controller-runtime Client] and gives you full control, but it comes with
122+
some drawbacks as well, because this method does not reflect directly real user workflows, and most importantly,
123+
the resulting tests are not as reusable with other infrastructure providers. (See [writing portable tests](#writing-portable-e2e-tests)).
124+
125+
We recommend using the [ClusterTemplate method] and the [Apply method] for creating objects in the cluster.
126+
This methods mimics the recommended user workflows, and it is based on `cluster-templates.yaml` files that can be
127+
provided via the [E2E config file], and thus easily swappable when changing the target infrastructure provider.
128+
129+
<aside class="note">
130+
131+
<h1>Tips</h1>
132+
133+
If you need control over object creation but you want to preserve portability, you can create many templates
134+
files each one creating only a small set of objects (instead of using a single template creating a full cluster).
135+
136+
</aside>
137+
138+
After creating objects in the cluster, use the existing methods in the [Cluster API test framework] to discover
139+
which object was created in the cluster so your code can adapt to different `cluster-templates.yaml` files.
140+
141+
Once you have objects references, the framework includes methods for waiting for the corresponding
142+
infrastructure to be provisioned, e.g. [WaitForClusterToProvision], [WaitForKubeadmControlPlaneMachinesToExist].
143+
144+
### Exec operations
145+
146+
You can use [Cluster API test framework] methods to modify Cluster API objects, as a last option, use
147+
the [controller-runtime Client].
148+
149+
The [Cluster API test framework] includes also methods for executing clusterctl operations, like e.g.
150+
the [ClusterTemplate method], the [ClusterctlMove method] etc.; in order to improve observability,
151+
each clusterctl operation creates a detailed log.
152+
153+
After using clusterctl operations, you can rely on the `Get` and on the `Wait` methods
154+
defined in the [Cluster API test framework] to check if the operation completed successfully.
155+
156+
## Tear down
157+
158+
After a test completes/fails, it is required to:
159+
160+
- Collect all the logs for the Cluster API controllers
161+
- Dump all the relevant Cluster API/Kubernetes objects
162+
- Cleanup all the infrastructure resources created during the test
163+
164+
Those task are usually implemented in the `AfterSuite`, and again the [Cluster API test framework] provides
165+
you useful methods for those tasks.
166+
167+
Please note that despite the fact that test specs are expected to delete objects in the management cluster and
168+
wait for the corresponding infrastructure to be terminated, it can happen that the test spec
169+
fails before starting object deletion or that objects deletion itself fails.
170+
171+
As a consequence, when scheduling/running a test suite, it is required to ensure all the generated
172+
resources are cleaned up. In Kubernetes, this is implemented by the [boskos] project.
173+
174+
## Writing portable E2E tests
175+
176+
A portable E2E test is a test can run with different infrastructure providers by simply
177+
changing the test configuration file.
178+
179+
Following recommendations should be followed to write portable E2E tests:
180+
181+
- Create different [E2E config file], one for each target infrastructure provider,
182+
providing different sets of env variables and timeout intervals.
183+
- Use the [InitManagementCluster method] for setting up the management cluster.
184+
- Use the [ClusterTemplate method] and the [Apply method]
185+
for creating objects in the cluster using `cluster-templates.yaml` files instead
186+
of hard coding object creation.
187+
- Use the `Get` methods defined in the [Cluster API test framework] to checks object
188+
being created, so your code can adapt to different `cluster-templates.yaml` files.
189+
- Never hard code the infrastructure provider name in your test spec.
190+
Instead, use the [InfrastructureProvider method] to get access to the
191+
name of the infrastructure provider defined in the [E2E config file].
192+
- Never hard code wait intervals in your test spec.
193+
Instead use the [GetIntervals method] to get access to the
194+
intervals defined in the [E2E config file].
195+
196+
## Cluster API conformance tests
197+
198+
As of today there is no a well-defined suites of E2E tests that can be used as a
199+
baseline for Cluster API conformance.
200+
201+
However, creating such suite is something that can provide a huge value for the
202+
long term success of the project.
203+
204+
The [test E2E package] provide examples of how this can be achieved implemeting a set of and reusable
205+
test specs for the most common Cluster API use cases.
206+
207+
<!-- links -->
208+
[Cluster API quick start]: https://cluster-api.sigs.k8s.io/user/quick-start.html
209+
[Cluster API test framework]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework?tab=doc
210+
[deprecated E2E config file]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework?tab=doc#Config
211+
[deprecated InitManagementCluster method]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework?tab=doc#InitManagementCluster
212+
[Apply method]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework?tab=doc#Applier
213+
[CAPA E2E tests]: https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/master/scripts/ci-e2e.sh
214+
[CAPG E2E tests]: https://github.com/kubernetes-sigs/cluster-api-provider-gcp/blob/master/scripts/ci-e2e.sh
215+
[WaitForClusterToProvision]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework?tab=doc#WaitForClusterToProvision
216+
[WaitForKubeadmControlPlaneMachinesToExist]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework?tab=doc#WaitForKubeadmControlPlaneMachinesToExist
217+
[controller-runtime Client]: https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.5.2/pkg/client?tab=doc#Client
218+
[boskos]: https://github.com/kubernetes/test-infra/tree/master/boskos
219+
[E2E config file]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework/clusterctl?tab=doc#E2EConfig
220+
[example E2E config file]: https://github.com/kubernetes-sigs/cluster-api/blob/master/test/e2e/config/docker-dev.yaml
221+
[NewKindClusterProvider]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework/bootstrap?tab=doc#NewKindClusterProvider
222+
[InitManagementClusterAndWatchControllerLogs method]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework/clusterctl?tab=doc#InitManagementClusterAndWatchControllerLogs
223+
[ClusterTemplate method]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework/clusterctl?tab=doc#ConfigCluster
224+
[ClusterctlMove method]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework/clusterctl?tab=doc#Move
225+
[InfrastructureProvider method]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework/clusterctl?tab=doc#E2EConfig.InfrastructureProviders
226+
[GetIntervals method]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework/clusterctl?tab=doc#E2EConfig.GetIntervals
227+
[test E2E package]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/e2e?tab=doc
228+
[CreateNamespaceAndWatchEvents method]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework?tab=doc#CreateNamespaceAndWatchEvents

‎test/framework/README.md‎

Lines changed: 1 addition & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -2,40 +2,4 @@
22

33
This framework aims to define common end-to-end patterns that can be reused across Cluster API providers.
44

5-
## Usage
6-
7-
Use this framework as you see fit. If there are pieces that don't work for you, do not use them. If there are pieces
8-
that almost work for you, customize them. If this does not fit your e2e testing please file an issue and we can discuss
9-
your use case and find a nice solution for everyone.
10-
11-
### Features
12-
13-
#### Optionally override the images that get loaded onto the management cluster.
14-
15-
This feature allows you to obtain a CAPI management image locally and use that specific image in your kind cluster.
16-
If you do not have one locally then the latest :master image will be used on the management cluster.
17-
18-
### Contents
19-
20-
This framework contains
21-
22-
* A [Go interface][mgmt] to a management cluster
23-
* A [struct that implements][impl] the management cluster interface using `kind` as the backend.
24-
* A series of behavioral tests
25-
26-
[mgmt]: ./interfaces.go
27-
[impl]: ./management/kind/mgmt.go
28-
29-
## Requirements
30-
31-
### Code
32-
33-
* You must use [ginkgo][ginkgo] for your testing framework.
34-
35-
[ginkgo]: https://onsi.github.io/ginkgo/
36-
37-
## Examples
38-
39-
To see this framework in use please take a look at the [docker provider found in the cluster-api repository][capd].
40-
41-
[capd]: https://github.com/kubernetes-sigs/cluster-api/tree/master/test/infrastructure/docker
5+
See https://cluster-api.sigs.k8s.io/developer/e2e.html for more information.

0 commit comments

Comments
 (0)