Skip to content

[rel-0.57] Trigger removal of CA tainted machine without waiting for replica change #976

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
aaronfern merged 1 commit into from
Mar 21, 2025
Merged

[rel-0.57] Trigger removal of CA tainted machine without waiting for replica change #976

aaronfern merged 1 commit into from
Mar 21, 2025

Conversation

@aaronfern
Copy link
Member

@aaronfern aaronfern commented Mar 21, 2025 •
edited
Loading

What this PR does / why we need it:
This PR cherry picks #972

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Release note:

CA tainted node is removed as soon as possible by MachineSet controller

@elankath @lenkite @aaronfern
Trigger removal of CA tainted machine without waiting for replica cha…
d37c4c1 
…nge (gardener#972)

* for-deletion machines dont want for replica change

* fix timing issue in additional unit test

* added unit test for managereplicas

* fixed ineffectual assignment&unused

* added additional check in test

* fixed test assertion

* addressed review comments

* accepted review comment

---------

Co-authored-by: elankath <tarun@elankath.com>
@aaronfern aaronfern requested a review from a team as a code owner March 21, 2025 06:14
@gardener-robot gardener-robot added needs/review Needs review size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 21, 2025
@gardener-robot-ci-1 gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) needs/ok-to-test Needs approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Mar 21, 2025
@aaronfern aaronfern merged commit e7f0f07 into gardener:rel-v0.57 Mar 21, 2025
8 checks passed
@gardener-robot gardener-robot added the status/closed Issue is closed (either delivered or triaged) label Mar 21, 2025
@aaronfern
Copy link
Member Author

aaronfern commented Mar 21, 2025

IT has passed

Starting integration tests...
Running Suite: Controller Suite - /Users/i544000/go/src/github.com/gardener/machine-controller-manager-provider-aws/test/integration/controller
===============================================================================================================================================
Random Seed: 1742551139

Will run 10 of 10 specs
------------------------------
[BeforeSuite]
/Users/i544000/go/src/github.com/gardener/machine-controller-manager-provider-aws/test/integration/controller/controller_test.go:47
  > Enter [BeforeSuite] TOP-LEVEL @ 03/21/25 15:29:30.316
  STEP: Checking for the clusters if provided are available @ 03/21/25 15:29:30.317
  2025/03/21 15:29:30 Control cluster kube-config - /Users/i544000/go/src/github.com/gardener/machine-controller-manager-provider-aws/dev/kube-configs/kubeconfig_control.yaml
  2025/03/21 15:29:30 Target cluster kube-config  - /Users/i544000/go/src/github.com/gardener/machine-controller-manager-provider-aws/dev/kube-configs/kubeconfig_target.yaml
  STEP: Checking Machine-Controller-Manager repo is available at: ../../../dev/mcm @ 03/21/25 15:29:34.16
  STEP: Scaledown existing machine controllers @ 03/21/25 15:29:34.16
  STEP: Starting Machine Controller  @ 03/21/25 15:29:34.335
  STEP: Starting Machine Controller Manager @ 03/21/25 15:29:34.35
  STEP: Setup MachineClass @ 03/21/25 15:29:34.359
  STEP: Looking for machineclass resource in the control cluster @ 03/21/25 15:29:35.552
  STEP: Looking for secrets refered in machineclass in the control cluster @ 03/21/25 15:29:35.718
  STEP: Initializing orphan resource tracker @ 03/21/25 15:29:36.054
  2025/03/21 15:29:38 orphan resource tracker initialized
  < Exit [BeforeSuite] TOP-LEVEL @ 03/21/25 15:29:38.345 (8.028s)
[BeforeSuite] PASSED [8.028 seconds]
------------------------------
Machine controllers test machine resource creation should not lead to any errors and add 1 more node in target cluster
/Users/i544000/go/src/github.com/gardener/machine-controller-manager/pkg/test/integration/common/framework.go:872
  > Enter [BeforeEach] Machine controllers test @ 03/21/25 15:29:38.345
  STEP: Checking machineController process is running @ 03/21/25 15:29:38.345
  STEP: Checking machineControllerManager process is running @ 03/21/25 15:29:38.345
  STEP: Checking nodes in target cluster are healthy @ 03/21/25 15:29:38.345
  < Exit [BeforeEach] Machine controllers test @ 03/21/25 15:29:38.896 (551ms)
  > Enter [It] should not lead to any errors and add 1 more node in target cluster @ 03/21/25 15:29:38.896
  STEP: Checking for errors @ 03/21/25 15:29:39.081
  STEP: Waiting until number of ready nodes is 1 more than initial nodes @ 03/21/25 15:29:39.252
  < Exit [It] should not lead to any errors and add 1 more node in target cluster @ 03/21/25 15:31:23.972 (1m45.077s)
• [105.628 seconds]
------------------------------
Machine controllers test machine resource deletion when machines available should not lead to errors and remove 1 node in target cluster
/Users/i544000/go/src/github.com/gardener/machine-controller-manager/pkg/test/integration/common/framework.go:895
  > Enter [BeforeEach] Machine controllers test @ 03/21/25 15:31:23.973
  STEP: Checking machineController process is running @ 03/21/25 15:31:23.973
  STEP: Checking machineControllerManager process is running @ 03/21/25 15:31:23.973
  STEP: Checking nodes in target cluster are healthy @ 03/21/25 15:31:23.973
  < Exit [BeforeEach] Machine controllers test @ 03/21/25 15:31:24.346 (373ms)
  > Enter [It] should not lead to errors and remove 1 node in target cluster @ 03/21/25 15:31:24.346
  STEP: Checking for errors @ 03/21/25 15:31:25.804
  STEP: Waiting until test-machine machine object is deleted @ 03/21/25 15:31:25.993
  STEP: Waiting until number of ready nodes is equal to number of initial  nodes @ 03/21/25 15:31:50.175
  < Exit [It] should not lead to errors and remove 1 node in target cluster @ 03/21/25 15:31:50.731 (26.385s)
• [26.758 seconds]
------------------------------
Machine controllers test machine resource deletion when machines are not available should keep nodes intact
/Users/i544000/go/src/github.com/gardener/machine-controller-manager/pkg/test/integration/common/framework.go:934
  > Enter [BeforeEach] Machine controllers test @ 03/21/25 15:31:50.731
  STEP: Checking machineController process is running @ 03/21/25 15:31:50.732
  STEP: Checking machineControllerManager process is running @ 03/21/25 15:31:50.732
  STEP: Checking nodes in target cluster are healthy @ 03/21/25 15:31:50.732
  < Exit [BeforeEach] Machine controllers test @ 03/21/25 15:31:51.105 (373ms)
  > Enter [It] should keep nodes intact @ 03/21/25 15:31:51.105
  STEP: Skipping as there are machines available and this check can't be performed @ 03/21/25 15:31:51.275
  < Exit [It] should keep nodes intact @ 03/21/25 15:31:51.275 (171ms)
• [0.544 seconds]
------------------------------
Machine controllers test machine deployment resource creation with replicas=0, scale up with replicas=1 should not lead to errors and add 1 more node to target cluster
/Users/i544000/go/src/github.com/gardener/machine-controller-manager/pkg/test/integration/common/framework.go:961
  > Enter [BeforeEach] Machine controllers test @ 03/21/25 15:31:51.275
  STEP: Checking machineController process is running @ 03/21/25 15:31:51.275
  STEP: Checking machineControllerManager process is running @ 03/21/25 15:31:51.275
  STEP: Checking nodes in target cluster are healthy @ 03/21/25 15:31:51.276
  < Exit [BeforeEach] Machine controllers test @ 03/21/25 15:31:51.651 (375ms)
  > Enter [It] should not lead to errors and add 1 more node to target cluster @ 03/21/25 15:31:51.651
  STEP: Checking for errors @ 03/21/25 15:31:51.838
  STEP: Waiting for Machine Set to be created @ 03/21/25 15:31:52.021
  STEP: Updating machineDeployment replicas to 1 @ 03/21/25 15:31:54.369
  STEP: Checking if machineDeployment's status has been updated with correct conditions @ 03/21/25 15:31:54.717
  STEP: Checking number of ready nodes==1 @ 03/21/25 15:33:58.993
  < Exit [It] should not lead to errors and add 1 more node to target cluster @ 03/21/25 15:34:00.108 (2m8.457s)
• [128.832 seconds]
------------------------------
Machine controllers test machine deployment resource scale-up with replicas=6 should not lead to errors and add further 5 nodes to target cluster
/Users/i544000/go/src/github.com/gardener/machine-controller-manager/pkg/test/integration/common/framework.go:1027
  > Enter [BeforeEach] Machine controllers test @ 03/21/25 15:34:00.109
  STEP: Checking machineController process is running @ 03/21/25 15:34:00.109
  STEP: Checking machineControllerManager process is running @ 03/21/25 15:34:00.109
  STEP: Checking nodes in target cluster are healthy @ 03/21/25 15:34:00.109
  < Exit [BeforeEach] Machine controllers test @ 03/21/25 15:34:00.481 (372ms)
  > Enter [It] should not lead to errors and add further 5 nodes to target cluster @ 03/21/25 15:34:00.481
  STEP: Checking for errors @ 03/21/25 15:34:00.83
  STEP: Checking number of ready nodes are 6 more than initial @ 03/21/25 15:34:00.83
  < Exit [It] should not lead to errors and add further 5 nodes to target cluster @ 03/21/25 15:35:45.513 (1m45.032s)
• [105.405 seconds]
------------------------------
Machine controllers test machine deployment resource scale-down with replicas=2 should not lead to errors and remove 4 nodes from target cluster
/Users/i544000/go/src/github.com/gardener/machine-controller-manager/pkg/test/integration/common/framework.go:1057
  > Enter [BeforeEach] Machine controllers test @ 03/21/25 15:35:45.513
  STEP: Checking machineController process is running @ 03/21/25 15:35:45.513
  STEP: Checking machineControllerManager process is running @ 03/21/25 15:35:45.513
  STEP: Checking nodes in target cluster are healthy @ 03/21/25 15:35:45.513
  < Exit [BeforeEach] Machine controllers test @ 03/21/25 15:35:45.889 (376ms)
  > Enter [It] should not lead to errors and remove 4 nodes from target cluster @ 03/21/25 15:35:45.889
  STEP: Checking for errors @ 03/21/25 15:35:47.295
  STEP: Checking number of ready nodes are 2 more than initial @ 03/21/25 15:35:47.295
  < Exit [It] should not lead to errors and remove 4 nodes from target cluster @ 03/21/25 15:36:43.958 (58.069s)
• [58.445 seconds]
------------------------------
Machine controllers test machine deployment resource scale-down with replicas=2 should freeze and unfreeze machineset temporarily
/Users/i544000/go/src/github.com/gardener/machine-controller-manager/pkg/test/integration/common/framework.go:1086
  > Enter [BeforeEach] Machine controllers test @ 03/21/25 15:36:43.958
  STEP: Checking machineController process is running @ 03/21/25 15:36:43.958
  STEP: Checking machineControllerManager process is running @ 03/21/25 15:36:43.959
  STEP: Checking nodes in target cluster are healthy @ 03/21/25 15:36:43.959
  < Exit [BeforeEach] Machine controllers test @ 03/21/25 15:36:44.327 (369ms)
  > Enter [It] should freeze and unfreeze machineset temporarily @ 03/21/25 15:36:44.328
  STEP: Searching for Froze in mcm log file @ 03/21/25 15:36:44.328
  STEP: Searching Unfroze in mcm log file @ 03/21/25 15:36:44.339
  < Exit [It] should freeze and unfreeze machineset temporarily @ 03/21/25 15:36:44.34 (13ms)
• [0.382 seconds]
------------------------------
Machine controllers test machine deployment resource updation to v2 machine-class and replicas=4 should upgrade machines and add more nodes to target
/Users/i544000/go/src/github.com/gardener/machine-controller-manager/pkg/test/integration/common/framework.go:1154
  > Enter [BeforeEach] Machine controllers test @ 03/21/25 15:36:44.34
  STEP: Checking machineController process is running @ 03/21/25 15:36:44.341
  STEP: Checking machineControllerManager process is running @ 03/21/25 15:36:44.341
  STEP: Checking nodes in target cluster are healthy @ 03/21/25 15:36:44.341
  < Exit [BeforeEach] Machine controllers test @ 03/21/25 15:36:44.715 (375ms)
  > Enter [It] should upgrade machines and add more nodes to target @ 03/21/25 15:36:44.715
  STEP: Checking for errors @ 03/21/25 15:36:45.068
  STEP: UpdatedReplicas to be 4 @ 03/21/25 15:36:45.068
  STEP: AvailableReplicas to be 4 @ 03/21/25 15:36:51.707
  STEP: Number of ready nodes be 4 more @ 03/21/25 15:38:46.52
  < Exit [It] should upgrade machines and add more nodes to target @ 03/21/25 15:39:43.716 (2m59.016s)
• [179.391 seconds]
------------------------------
Machine controllers test machine deployment resource deletion When there are machine deployment(s) available in control cluster should not lead to errors and list only initial nodes
/Users/i544000/go/src/github.com/gardener/machine-controller-manager/pkg/test/integration/common/framework.go:1208
  > Enter [BeforeEach] Machine controllers test @ 03/21/25 15:39:43.716
  STEP: Checking machineController process is running @ 03/21/25 15:39:43.717
  STEP: Checking machineControllerManager process is running @ 03/21/25 15:39:43.717
  STEP: Checking nodes in target cluster are healthy @ 03/21/25 15:39:43.717
  < Exit [BeforeEach] Machine controllers test @ 03/21/25 15:39:44.092 (376ms)
  > Enter [It] should not lead to errors and list only initial nodes @ 03/21/25 15:39:44.092
  STEP: Checking for errors @ 03/21/25 15:39:44.25
  STEP: Waiting until number of ready nodes is equal to number of initial  nodes @ 03/21/25 15:39:44.415
  < Exit [It] should not lead to errors and list only initial nodes @ 03/21/25 15:40:34.552 (50.46s)
• [50.836 seconds]
------------------------------
Machine controllers test orphaned resources when the hyperscaler resources are queried should have been deleted
/Users/i544000/go/src/github.com/gardener/machine-controller-manager/pkg/test/integration/common/framework.go:1245
  > Enter [BeforeEach] Machine controllers test @ 03/21/25 15:40:34.553
  STEP: Checking machineController process is running @ 03/21/25 15:40:34.553
  STEP: Checking machineControllerManager process is running @ 03/21/25 15:40:34.553
  STEP: Checking nodes in target cluster are healthy @ 03/21/25 15:40:34.553
  < Exit [BeforeEach] Machine controllers test @ 03/21/25 15:40:34.925 (372ms)
  > Enter [It] should have been deleted @ 03/21/25 15:40:34.925
  STEP: Querying and comparing @ 03/21/25 15:40:34.925
  < Exit [It] should have been deleted @ 03/21/25 15:40:36.966 (2.041s)
• [2.413 seconds]
------------------------------
[AfterSuite]
/Users/i544000/go/src/github.com/gardener/machine-controller-manager-provider-aws/test/integration/controller/controller_test.go:49
  > Enter [AfterSuite] TOP-LEVEL @ 03/21/25 15:40:36.966
  STEP: Running Cleanup @ 03/21/25 15:40:36.966
  2025/03/21 15:40:57 machinedeployments.machine.sapcloud.io "test-machine-deployment" not found
  2025/03/21 15:40:57 machines.machine.sapcloud.io "test-machine" not found
  2025/03/21 15:40:57 deleting test-mc-v1 machineclass
  2025/03/21 15:40:58 machineclass deleted
  2025/03/21 15:40:58 deleting test-mc-v2 machineclass
  2025/03/21 15:40:59 machineclass deleted
  < Exit [AfterSuite] TOP-LEVEL @ 03/21/25 15:40:59.85 (22.884s)
[AfterSuite] PASSED [22.885 seconds]
------------------------------

Ran 10 of 10 Specs in 689.548 seconds
SUCCESS! -- 10 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS

Ginkgo ran 1 suite in 12m0.193426375s
Test Suite Passed
Integration tests completed successfully

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@takoverflow takoverflow takoverflow approved these changes

Assignees

No one assigned

Labels

needs/ok-to-test Needs approval for testing (check PR in detail before setting this label because PR is run on CI/CD) needs/review Needs review size/M Denotes a PR that changes 30-99 lines, ignoring generated files. status/closed Issue is closed (either delivered or triaged)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@aaronfern @takoverflow @gardener-robot-ci-1 @gardener-robot @elankath