e2e: Add kubernetes version upgrade test #141
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
k8s-ci-robot
added
do-not-merge/work-in-progress
✅ Deploy Preview for kubernetes-sigs-cluster-api-cloudstack ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
k8s-ci-robot
added
approved
test/e2e/config/cloudstack.yaml
Outdated
| @@ -129,7 +132,8 @@ variables: | |||
| CLOUDSTACK_IMPOSSIBLE_CONTROL_PLANE_MACHINE_OFFERING: "Impossible Instance" | |||
| CLOUDSTACK_WORKER_MACHINE_OFFERING: "Medium Instance" | |||
| CLOUDSTACK_IMPOSSIBLE_WORKER_MACHINE_OFFERING: "Impossible Instance" | |||
| CLOUDSTACK_TEMPLATE_NAME: kube-v1.20.10/ubuntu-2004 | |||
| CLOUDSTACK_TEMPLATE_NAME: kube-v1.22.6/ubuntu-2004 | |||
There was a problem hiding this comment.
I believe these will break our current e2e, as we don't have this template on our ACS configuration. We'll need to adapt it accordingly.
There was a problem hiding this comment.
New templates can be found here
http://packages.shapeblue.com/cluster-api-provider-cloudstack/images/
k8s-ci-robot
added
the
needs-rebase
davidjumani
force-pushed
the
add-kube-upgrade-test
branch
from
e4f52d2 to
f9599c9
Compare
k8s-ci-robot
removed
the
needs-rebase
Codecov Report
@@ Coverage Diff @@
## main #141 +/- ##
=======================================
Coverage 32.97% 32.97%
=======================================
Files 43 43
Lines 3733 3733
=======================================
Hits 1231 1231
Misses 2340 2340
Partials 162 162 Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
k8s-ci-robot
removed
the
do-not-merge/work-in-progress
davidjumani
force-pushed
the
add-kube-upgrade-test
branch
from
f9599c9 to
e791b1f
Compare
rejoshed
reviewed
Sep 12, 2022
There was a problem hiding this comment.
New comments.
Updating a cloudstackmachine template in fact does not cause a rolling update.
The only reason it appeared to is that the k8s version in the kubeadmcontrolplane was bumped as well. Modifying the cloudstackmachinetemplate alone does not trigger this.
As such, it is incorrect to allow modifications on cloudstackmachinetemplates.
If modifications are allowed, then a VM could be out of spec.
Further, the intent/design by the CAPI authors is for a new cloudstackmachinetemplate to be created under a new ref, and the previous to be deleted.
See here for further documentation:
We will be reverting the changes to the cloudstackmachinetemplate update verification webhooks to once again enforce the immutability of cloudstackmachinetemplates.
Please update this test to use the new method wherein a new cloudstackmachine template is created and reffered to.
Thank you and I will try the E2E test again once this has been updated.
Previous comments.
I have seen that many of the VMs bump to 1.23.x when running this test after installing the two new VM templates in ACS, but I'm still seeing the following issue, and not all machines upgrade themselves.
Can you please investigate or point me to where to investigate. I'd like to make this run so I can merge it. Thanks!
STEP: Dumping logs from the "kubernetes-version-upgrade-bic3xv" workload cluster
Failed to get logs for machine kubernetes-version-upgrade-bic3xv-control-plane-wk9zj, cluster kubernetes-version-upgrade-vwc1n0/kubernetes-version-upgrade-bic3xv: error creating container exec: Error response from daemon: No such container: kubernetes-version-upgrade-bic3xv-control-plane-wk9zj
Failed to get logs for machine kubernetes-version-upgrade-bic3xv-md-0-655b47fcfd-brnzx, cluster kubernetes-version-upgrade-vwc1n0/kubernetes-version-upgrade-bic3xv: error creating container exec: Error response from daemon: No such container: kubernetes-version-upgrade-bic3xv-md-0-655b47fcfd-brnzx
STEP: Dumping all the Cluster API resources in the "kubernetes-version-upgrade-vwc1n0" namespace
STEP: Deleting cluster kubernetes-version-upgrade-vwc1n0/kubernetes-version-upgrade-bic3xv
STEP: Deleting cluster kubernetes-version-upgrade-bic3xv
INFO: Waiting for the Cluster kubernetes-version-upgrade-vwc1n0/kubernetes-version-upgrade-bic3xv to be deleted
STEP: Waiting for cluster kubernetes-version-upgrade-bic3xv to be deleted
STEP: Deleting namespace used for hosting the "kubernetes-version-upgrade" test spec
INFO: Deleting namespace kubernetes-version-upgrade-vwc1n0
• Failure [476.556 seconds]
When testing Kubernetes version upgrades
/Users/rejoshed/code/cluster-api-provider-cloudstack/test/e2e/kubernetes_version_upgrade_test.go:23
Should successfully upgrade kubernetes versions when there is a change in relevant fields [It]
/Users/rejoshed/code/cluster-api-provider-cloudstack/test/e2e/kubernetes_version_upgrade.go:61
Unexpected error:
<*errors.withStack | 0xc002686c78>: {
error: <*exec.ExitError | 0xc00076e3c0>{
ProcessState: {
pid: 5184,
status: 256,
rusage: {
Utime: {
Sec: 0,
Usec: 44177,
Pad_cgo_0: [0, 0, 0, 0],
},
Stime: {
Sec: 0,
Usec: 90809,
Pad_cgo_0: [0, 0, 0, 0],
},
Maxrss: 26382336,
Ixrss: 0,
Idrss: 0,
Isrss: 0,
Minflt: 3484,
Majflt: 3251,
Nswap: 0,
Inblock: 0,
Oublock: 0,
Msgsnd: 1,
Msgrcv: 0,
Nsignals: 86,
Nvcsw: 29,
Nivcsw: 1668,
},
},
Stderr: nil,
},
stack: [0x2536740, 0x2536c90, 0x261378c, 0x2628be2, 0x2951a48, 0x139d5d1, 0x139cfc5, 0x139c6bb, 0x13a23aa, 0x13a1da7, 0x13c22e8, 0x13c2005, 0x13c16a5, 0x13c3a72, 0x13cfb69, 0x13cf976, 0x295bfea, 0x11196a2, 0x106a961],
}
exit status 1
occurred
/Users/rejoshed/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.1.0/framework/clusterctl/clusterctl_helpers.go:300
Full Stack Trace
sigs.k8s.io/cluster-api/test/framework/clusterctl.ApplyClusterTemplateAndWait({_, _}, {{0x3329c30, 0xc001ba56c0}, {{0xc0020cc480, 0x52}, {0xc000cc0875, 0x61}, {0xc000cc08d7, 0x1c}, ...}, ...}, ...)
/Users/rejoshed/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.1.0/framework/clusterctl/clusterctl_helpers.go:300 +0xc35
sigs.k8s.io/cluster-api-provider-cloudstack-staging/test/e2e.KubernetesVersionUpgradeSpec.func2()
/Users/rejoshed/code/cluster-api-provider-cloudstack/test/e2e/kubernetes_version_upgrade.go:63 +0x4e8
github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0x0?)
/Users/rejoshed/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/leafnodes/runner.go:113 +0xb1
github.com/onsi/ginkgo/internal/leafnodes.(*runner).run(0x0?)
/Users/rejoshed/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/leafnodes/runner.go:64 +0x125
github.com/onsi/ginkgo/internal/leafnodes.(*ItNode).Run(0xc00082e480?)
/Users/rejoshed/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/leafnodes/it_node.go:26 +0x7b
github.com/onsi/ginkgo/internal/spec.(*Spec).runSample(0xc00087af00, 0xc001cd79f0?, {0x3301060, 0xc0001788c0})
/Users/rejoshed/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/spec/spec.go:215 +0x28a
github.com/onsi/ginkgo/internal/spec.(*Spec).Run(0xc00087af00, {0x3301060, 0xc0001788c0})
/Users/rejoshed/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/spec/spec.go:138 +0xe7
github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpec(0xc0004402c0, 0xc00087af00)
/Users/rejoshed/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/specrunner/spec_runner.go:200 +0xe8
github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpecs(0xc0004402c0)
/Users/rejoshed/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/specrunner/spec_runner.go:170 +0x1a5
github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).Run(0xc0004402c0)
/Users/rejoshed/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/specrunner/spec_runner.go:66 +0xc5
github.com/onsi/ginkgo/internal/suite.(*Suite).Run(0xc000190b60, {0x6787c48, 0xc0004a9a00}, {0x2fd1192, 0x8}, {0xc000422a00, 0x2, 0x2}, {0x331fc50, 0xc0001788c0}, ...)
/Users/rejoshed/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/suite/suite.go:79 +0x4d2
github.com/onsi/ginkgo.runSpecsWithCustomReporters({0x3303aa0?, 0xc0004a9a00}, {0x2fd1192, 0x8}, {0xc0004229e0, 0x2, 0x2ff0fd0?})
/Users/rejoshed/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/ginkgo_dsl.go:245 +0x189
github.com/onsi/ginkgo.RunSpecsWithDefaultAndCustomReporters({0x3303aa0, 0xc0004a9a00}, {0x2fd1192, 0x8}, {0xc000092f50, 0x1, 0x1})
/Users/rejoshed/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/ginkgo_dsl.go:228 +0x1b6
sigs.k8s.io/cluster-api-provider-cloudstack-staging/test/e2e.TestE2E(0x0?)
/Users/rejoshed/code/cluster-api-provider-cloudstack/test/e2e/e2e_suite_test.go:97 +0x10a
testing.tRunner(0xc0004a9a00, 0x30dce98)
/usr/local/Cellar/go@1.18/1.18.5/libexec/src/testing/testing.go:1439 +0x102
created by testing.(*T).Run
/usr/local/Cellar/go@1.18/1.18.5/libexec/src/testing/testing.go:1486 +0x35f
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSTEP: Dumping logs from the bootstrap cluster
Failed to get logs for the bootstrap cluster node capi-test-control-plane: exit status 1
STEP: Tearing down the management cluster
JUnit report was created: /Users/rejoshed/code/cluster-api-provider-cloudstack/_artifacts/junit.e2e_suite.1.xml
Summarizing 1 Failure:
[Fail] When testing Kubernetes version upgrades [It] Should successfully upgrade kubernetes versions when there is a change in relevant fields
/Users/rejoshed/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.1.0/framework/clusterctl/clusterctl_helpers.go:300
Ran 1 of 28 Specs in 528.317 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 27 Skipped
--- FAIL: TestE2E (528.33s)
FAIL
|
@davidjumani do you still want to PR this? Will you have a chance to update the test to work with the machine template immutability changes? |
|
/lgtm |
k8s-ci-robot
added
the
lgtm
davidjumani
force-pushed
the
add-kube-upgrade-test
branch
from
e791b1f to
6c3e7ff
Compare
k8s-ci-robot
removed
the
lgtm
|
Tests passed |
k8s-ci-robot
added
the
needs-rebase
davidjumani
force-pushed
the
add-kube-upgrade-test
branch
from
6c3e7ff to
9b44240
Compare
k8s-ci-robot
removed
the
needs-rebase
davidjumani
force-pushed
the
add-kube-upgrade-test
branch
from
9b44240 to
a2eae1e
Compare
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: davidjumani, rohityadavcloud The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Tests were aborted. |
vignesh-goutham
pushed a commit
to vignesh-goutham/cluster-api-provider-cloudstack
that referenced
this pull request
Jul 12, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes:
Adds a Kubernetes version upgrade to the e2e test suite
Testing performed:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.