Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 2307909: Fix disable dr when VR failed validation #369

Merged

Conversation

nirs
Copy link

@nirs nirs commented Oct 2, 2024

When deleting a primary VRG, we wait until the VR Completed condition is
met. However if a VR precondition failed, for example using a drpolicy
without flattening enabled when the PVC needs flattening, the VR will
never complete and the vrg and drpc deletion will never complete.

Since csi-addons 0.10.0 we have a new Validated VR condition, set to
true if pre conditions are met, and false if not. VR is can be deleted
safely in this state, since mirroring was not enabled.

This changes modifies deleted VRG processing to check the new VR
Validated status. If the condition exist and the condition status is
false, validateVRStatus() return true, signaling that the VR is in the
desired state, and ramen completes the delete flow.

If the VR does not report the Validated condition (e.g. old csi-addon
version) or the condition status is true (mirroring in progress), we
continue in the normal flow. The VR will be deleted only when the
Completed condition status is true.

This is a backport of:

  1. Bump CSI version to 0.9.1 RamenDR/ramen#1542 - updating for API and dependencies changes in csi-addons 0.9.1
  2. Update csi-addons to v0.10.0 RamenDR/ramen#1561 - for getting csi-addons 0.10.0 in ramen and drenv
  3. Fix disable dr when VR failed validation RamenDR/ramen#1570 - using the new Validated condition in csi-addons 0.10.0

Tested with:

  • drenv with deployment using cloned pvc
  • drenv with vm using cloned pvc

Image for testing: quay.io/nirsof/ramen-operator:release-4.17-vr-validated-1

ELENAGER and others added 2 commits October 2, 2024 13:44
Signed-off-by: Elena Gershkovich <elenage@il.ibm.com>
(cherry picked from commit 1fbfec3)
This brings the new Validated condition needed for fixing disable dr
when a VR precondition has failed. With this change we can use the new
feature in ramen for testing the fix.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
(cherry picked from commit f888239)
Copy link

openshift-ci bot commented Oct 2, 2024

@nirs: This pull request references Bugzilla bug 2307909, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (ODF 4.17.0) matches configured target release for branch (ODF 4.17.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @keesturam

In response to this:

Bug 2307909: Fix disable dr when VR failed validation

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Copy link

openshift-ci bot commented Oct 2, 2024

@openshift-ci[bot]: GitHub didn't allow me to request PR reviews from the following users: keesturam.

Note that only red-hat-storage members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

@nirs: This pull request references Bugzilla bug 2307909, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (ODF 4.17.0) matches configured target release for branch (ODF 4.17.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @keesturam

In response to this:

Bug 2307909: Fix disable dr when VR failed validation

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@nirs
Copy link
Author

nirs commented Oct 2, 2024

@ELENAGER can you review?

nirs added 8 commits October 2, 2024 15:12
Done using:

    % go get github.com/csi-addons/kubernetes-csi-addons@v0.10.0
    go: upgraded github.com/csi-addons/kubernetes-csi-addons v0.9.1 => v0.10.0
    go: upgraded github.com/google/pprof v0.0.0-20240727154555-813a5fbdbec8 => v0.0.0-20240827171923-fa2c70bbbfe5
    go: upgraded github.com/onsi/ginkgo/v2 v2.20.0 => v2.20.2
    go: upgraded github.com/onsi/gomega v1.34.1 => v1.34.2
    go: upgraded golang.org/x/sys v0.23.0 => v0.24.0
    go: upgraded k8s.io/api v0.31.0 => v0.31.1
    go: upgraded k8s.io/apimachinery v0.31.0 => v0.31.1

    % go mod tidy

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
(cherry picked from commit c0e382d)
Was forgetten when we replaced minikube volumesnapshot addons.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
(cherry picked from commit 75aa24b)
The comment is still unclear, but at least easier to read.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
(cherry picked from commit c8b3512)
- Replace "VR under deletion" with "deleted VR", matching the
  terminology used in the code.
- Replace "detected as missing or deleted" with "is missing" when we
  know the VR does not exist.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
(cherry picked from commit b9552c3)
Moved from validateVRStatus() to make room from checking VR VAlidated
status. This also make the function easier to understand, keeping the
same level of abstraction and getting rid of the uninteresting details.

Conflicts:
- internal/controller/vrg_volrep.go - adapt to current code using
  volRep.Namespace/Name. In upstream we use pvc.Namespace/Name, added in
  RamenDR#1528

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
(cherry picked from commit bd41cab)
Rename msg to errorMsg and document that this is an error message,
if the we could not get the condition value, because it is missing,
stale, or unknown.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
(cherry picked from commit 830f770)
Log after important changes to the system in delete VR flow to make it
easier to understand what the system is doing, and how ramen changed the
system.

New logs:
- delete the VR resource
- remove annotations from PV

Improve logs:
- remove annotations, labels, and finalizers from PVC

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
(cherry picked from commit 7d170b5)
When deleting a primary VRG, we wait until the VR Completed condition is
met. However if a VR precondition failed, for example using a drpolicy
without flattening enabled when the PVC needs flattening, the VR will
never complete and the vrg and drpc deletion will never complete.

Since csi-addons 0.10.0 we have a new Validated VR condition, set to
true if pre conditions are met, and false if not. VR is can be deleted
safely in this state, since mirroring was not enabled.

This changes modifies deleted VRG processing to check the new VR
Validated status. If the condition exist and the condition status is
false, validateVRStatus() return true, signaling that the VR is in the
desired state, and ramen completes the delete flow.

If the VR does not report the Validated condition (e.g. old csi-addon
version) or the condition status is true (mirroring in progress), we
continue in the normal flow. The VR will be deleted only when the
Completed condition status is true.

Tested with discovered deployment and vm using a pvc created from a
volume snapshot.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>

Conflicts:
- internal/controller/vrg_volrep.go: adapt current code using
  volRep.Namespace/Name instead of pvc.Namespace/Name added in
  RamenDR#1528

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
(cherry picked from commit 5dfc74a)
@nirs nirs force-pushed the release-4.17-vr-validated branch from f98eb0f to 8689b8e Compare October 2, 2024 12:12
Copy link

openshift-ci bot commented Oct 2, 2024

@nirs: This pull request references Bugzilla bug 2307909, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (ODF 4.17.0) matches configured target release for branch (ODF 4.17.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @keesturam

In response to this:

Bug 2307909: Fix disable dr when VR failed validation

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Copy link

openshift-ci bot commented Oct 2, 2024

@openshift-ci[bot]: GitHub didn't allow me to request PR reviews from the following users: keesturam.

Note that only red-hat-storage members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

@nirs: This pull request references Bugzilla bug 2307909, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (ODF 4.17.0) matches configured target release for branch (ODF 4.17.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @keesturam

In response to this:

Bug 2307909: Fix disable dr when VR failed validation

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@nirs
Copy link
Author

nirs commented Oct 2, 2024

Unit tests failure is unrelated, but I cannot rerun the tests in this repo.

Copy link

openshift-ci bot commented Oct 2, 2024

@nirs: This pull request references Bugzilla bug 2307909, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (ODF 4.17.0) matches configured target release for branch (ODF 4.17.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @keesturam

In response to this:

Bug 2307909: Fix disable dr when VR failed validation

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Copy link

openshift-ci bot commented Oct 2, 2024

@openshift-ci[bot]: GitHub didn't allow me to request PR reviews from the following users: keesturam.

Note that only red-hat-storage members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

@nirs: This pull request references Bugzilla bug 2307909, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (ODF 4.17.0) matches configured target release for branch (ODF 4.17.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @keesturam

In response to this:

Bug 2307909: Fix disable dr when VR failed validation

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@agarwal-mudit
Copy link
Member

/cc @ELENAGER

Copy link

openshift-ci bot commented Oct 7, 2024

@agarwal-mudit: GitHub didn't allow me to request PR reviews from the following users: ELENAGER.

Note that only red-hat-storage members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @ELENAGER

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Copy link

openshift-ci bot commented Oct 7, 2024

@ELENAGER: changing LGTM is restricted to collaborators

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Copy link

openshift-ci bot commented Oct 7, 2024

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by: ELENAGER, nirs, ShyamsundarR

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copy link

openshift-ci bot commented Oct 7, 2024

@nirs: This pull request references Bugzilla bug 2307909, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (ODF 4.17.0) matches configured target release for branch (ODF 4.17.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @keesturam

In response to this:

Bug 2307909: Fix disable dr when VR failed validation

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Copy link

openshift-ci bot commented Oct 7, 2024

@openshift-ci[bot]: GitHub didn't allow me to request PR reviews from the following users: keesturam.

Note that only red-hat-storage members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

@nirs: This pull request references Bugzilla bug 2307909, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (ODF 4.17.0) matches configured target release for branch (ODF 4.17.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @keesturam

In response to this:

Bug 2307909: Fix disable dr when VR failed validation

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@nirs
Copy link
Author

nirs commented Oct 7, 2024

Tested also with vm.

@agarwal-mudit
Copy link
Member

Should we merge it manually?

@ShyamsundarR ShyamsundarR merged commit e35ac96 into red-hat-storage:release-4.17 Oct 7, 2024
14 of 16 checks passed
Copy link

openshift-ci bot commented Oct 7, 2024

@nirs: All pull requests linked via external trackers have merged:

Bugzilla bug 2307909 has been moved to the MODIFIED state.

In response to this:

Bug 2307909: Fix disable dr when VR failed validation

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants