OCPBUGS-47489: pkg/gcp/destroy: add waits to prevent leaks during heavy load #9384
Conversation
The GCP destroy code repeated a lot of boilerplate operation handling. This refactors all of that into a single function for increased maintainability.
|
Skipping CI for Draft Pull Request. |
openshift-ci
bot
added
the
do-not-merge/work-in-progress
openshift-ci-robot
added
jira/valid-reference
|
@patrickdillon: This pull request references Jira Issue OCPBUGS-47489, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci
bot
removed
the
do-not-merge/work-in-progress
|
/jira refresh |
openshift-ci-robot
added
jira/valid-bug
|
@patrickdillon: This pull request references Jira Issue OCPBUGS-47489, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
Requesting review from QA contact: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
In OCPBUGS-47489, we see that some resources, particularly global backend services are being leaked during the destroy process. Analysis of the creation time stamps for the leaked resources shows that the resources are clustered together, suggesting the leaks may occur during periods of heavy load. During periods of heavy load, the deletion may take longer to process. This commit addresses the issue by adding waits for all resource deletion. This ensures ample time to complete destroy calls.
patrickdillon
force-pushed
the
ocpbugs-47489-global-leak
branch
from
961b6a9 to
bd81ab8
Compare
patrickdillon
changed the title
|
@patrickdillon: An error was encountered adding this pull request to the external tracker bugs for bug OCPBUGS-47489 on the Jira server at https://issues.redhat.com/. No known errors were detected, please see the full error message for details. Full error message.
failed to update remote link: failed to update link: request failed. Please analyze the request body for more details. Status code: 403: {"errorMessages":["No Link Issue Permission for issue 'OCPBUGS-47489'."],"errors":{}}
Please contact an administrator to resolve this issue, then request a bug refresh with In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
patrickdillon
force-pushed
the
ocpbugs-47489-global-leak
branch
from
bd81ab8 to
82b994d
Compare
|
Pushed some changes to (hopefully) appease the linter. |
|
@patrickdillon: This pull request references Jira Issue OCPBUGS-47489. The bug has been updated to no longer refer to the pull request using the external bug tracker. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@patrickdillon: This pull request references Jira Issue OCPBUGS-47489, which is valid. 3 validation(s) were run on this bug
Requesting review from QA contact: The bug has been updated to refer to the pull request using the external bug tracker. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
Linter was complaining about existing code that used the magic string "DONE". This converts the string to a constant.
patrickdillon
force-pushed
the
ocpbugs-47489-global-leak
branch
from
82b994d to
c78c0a1
Compare
|
@patrickdillon: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/label acknowledge-critical-fixes-only |
openshift-ci
bot
added
the
acknowledge-critical-fixes-only
|
/approve |
|
@patrickdillon: once the present PR merges, I will cherry-pick it on top of In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: patrickdillon The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
openshift-merge-bot
bot
merged commit d224d2c
into
openshift:main
|
@patrickdillon: Jira Issue OCPBUGS-47489: All pull requests linked via external trackers have merged: Jira Issue OCPBUGS-47489 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@patrickdillon: new pull request created: #9402 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[ART PR BUILD NOTIFIER] Distgit: ose-installer-altinfra |
|
[ART PR BUILD NOTIFIER] Distgit: ose-installer-terraform-providers |
|
[ART PR BUILD NOTIFIER] Distgit: ose-baremetal-installer |
|
[ART PR BUILD NOTIFIER] Distgit: ose-installer-artifacts |
|
__ |
We're slowly leaking backend services. This PR adds waits to make sure the operation is done before proceeding. This refactors the destroy code to add a central operation handling function to facilitate this change.