Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[release-4.18] OCPBUGS-49320: pkg/gcp/destroy: add waits to prevent leaks during heavy load #9402

Open
wants to merge 3 commits into
base: release-4.18
Choose a base branch
from
Open

[release-4.18] OCPBUGS-49320: pkg/gcp/destroy: add waits to prevent leaks during heavy load #9402

wants to merge 3 commits into from

Conversation

openshift-cherrypick-robot
Copy link

This is an automated cherry-pick of #9384

/assign patrickdillon

@patrickdillon
pkg/destroy/gcp: refactor operation handling
261dbcc 
The GCP destroy code repeated a lot of boilerplate operation handling.
This refactors all of that into a single function for increased
maintainability.
@patrickdillon
OCPBUGS-47489: wait for destroy operations
9305d23 
In OCPBUGS-47489, we see that some resources, particularly global
backend services are being leaked during the destroy process.
Analysis of the creation time stamps for the leaked resources shows
that the resources are clustered together, suggesting the leaks
may occur during periods of heavy load.

During periods of heavy load, the deletion may take longer to
process. This commit addresses the issue by adding waits for all
resource deletion. This ensures ample time to complete destroy calls.
@patrickdillon
pkg/destroy/gcp: add DONE const to appease linter
e0f34dd 
Linter was complaining about existing code that used the magic
string "DONE". This converts the string to a constant.
@openshift-cherrypick-robot openshift-cherrypick-robot mentioned this pull request Jan 25, 2025
Merged
@openshift-ci-robot
Copy link
Contributor

@openshift-cherrypick-robot: Jira Issue OCPBUGS-47489 has been cloned as Jira Issue OCPBUGS-49320. Will retitle bug to link to clone.
/retitle [release-4.18] OCPBUGS-49320: pkg/gcp/destroy: add waits to prevent leaks during heavy load

In response to this:

This is an automated cherry-pick of #9384

/assign patrickdillon

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot changed the title [release-4.18] OCPBUGS-47489: pkg/gcp/destroy: add waits to prevent leaks during heavy load [release-4.18] OCPBUGS-49320: pkg/gcp/destroy: add waits to prevent leaks during heavy load Jan 25, 2025
@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jan 25, 2025
@openshift-ci-robot
Copy link
Contributor

@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-49320, which is invalid:

  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
  • expected dependent Jira Issue OCPBUGS-47489 to target a version in 4.19.0, but it targets "4.19" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

This is an automated cherry-pick of #9384

/assign patrickdillon

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Jan 25, 2025 •
edited
Loading

@openshift-cherrypick-robot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-gcp-default-config e0f34dd link false /test e2e-gcp-default-config
ci/prow/e2e-gcp-ovn-byo-vpc e0f34dd link false /test e2e-gcp-ovn-byo-vpc
ci/prow/e2e-gcp-ovn-xpn e0f34dd link false /test e2e-gcp-ovn-xpn
ci/prow/e2e-gcp-secureboot e0f34dd link false /test e2e-gcp-secureboot

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@sadasu
Copy link
Contributor

sadasu commented Jan 28, 2025

/retest-required

@gpei
Copy link
Contributor

gpei commented Jan 29, 2025

/jira refresh

@openshift-ci-robot
Copy link
Contributor

@gpei: This pull request references Jira Issue OCPBUGS-49320, which is invalid:

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@gpei
Copy link
Contributor

gpei commented Jan 29, 2025

/label cherry-pick-approved

@openshift-ci openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Jan 29, 2025
@gpei
Copy link
Contributor

gpei commented Jan 29, 2025

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jan 29, 2025
@openshift-ci-robot
Copy link
Contributor

@gpei: This pull request references Jira Issue OCPBUGS-49320, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.18.0) matches configured target version for branch (4.18.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-47489 is in the state Verified, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
  • dependent Jira Issue OCPBUGS-47489 targets the "4.19.0" version, which is one of the valid target versions: 4.19.0
  • bug has dependents

Requesting review from QA contact:
/cc @gpei

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot requested a review from gpei January 29, 2025 06:19
@sadasu
Copy link
Contributor

sadasu commented Jan 30, 2025

/label backport-risk-assessed

@sadasu
Copy link
Contributor

sadasu commented Jan 30, 2025

/approve

@openshift-ci openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Jan 30, 2025
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Jan 30, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sadasu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jhixson74 jhixson74 Awaiting requested review from jhixson74

@patrickdillon patrickdillon Awaiting requested review from patrickdillon

@gpei gpei Awaiting requested review from gpei

Assignees

@patrickdillon patrickdillon

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@openshift-cherrypick-robot @openshift-ci-robot @sadasu @gpei @patrickdillon