WIP: Vr status test #1642
Closed
WIP: Vr status test #1642
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Use a development build of csi-addons adding .Message to the all conditions: csi-addons/kubernetes-csi-addons#691 This is only needed for testing, I'll remove this before merging. Signed-off-by: Nir Soffer <nsoffer@redhat.com>
It was added in failed state (Status=False, Reason=PrerequisiteNotMet) by default which is wrong. The issue was hidden since we check the condition only when deleting the VRG. We want to inspect the condition when the when the VRG is live, so we can report the condition status in the protected pvcs conditions. Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Based on the messages addded in csi-addons/kubernetes-csi-addons#691. We want to propagate the error messages to the protected pvcs conditions. Signed-off-by: Nir Soffer <nsoffer@redhat.com>
When a VR condition is not met, we set the protected PVC condition
message using the error message returned from isVRConditionMet(). When
using csi-addons > 0.10.0, we use now the message from the condition
instead of the default message.
Since the Validated condition is not reported by older version of
csi-addons, and we must wait until the Validated condition status is
known when VRG is deleted, isVRConditionMet() returns now also the state
of the condition, which can be:
- missing: condition not found
- stale: observed generation does not match object generation
- unknown: the special "Unknown" value
- known: status is True or False
When we validate the Validate condition we have these cases:
- Condition is missing: continue to next condition.
- Condition is met: continue to the next condition.
- Condition not met and its status is False. This VR will never
complete and it is safe to delete since replication will never start.
If VRG is deleted, we return true since the VR reached the designed
state. Otherwise we return false. In this case we updated the
protected pvc condition with the message from the VR condition.
- Condition is not met and is stale or unnown: we need to check again
later. There is no point to check the completed condition since a VR
cannot complete without validation.In this case we updated the
protected pvc condition with the message generated by
isVRConditionMet() for stale or unknown conditions.
Example protected pvc DataReady condition with propagated message when
VR validation failed:
conditions:
- lastTransitionTime: "2024-11-06T15:33:06Z"
message: 'failed to meet prerequisite: rpc error: code = FailedPrecondition
desc = system is not in a state required for the operation''s execution:
failed to enable mirroring on image "replicapool/csi-vol-fe2ca7f8-713c-4c51-bf52-0d4b2c11d329":
parent image "replicapool/csi-snap-e2114105-b451-469b-ad97-eb3cbe2af54e"
is not enabled for mirroring'
observedGeneration: 1
reason: Error
status: "False"
type: DataReady
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Verify that when VR Validated condition message is propagated to the
protected pvc condition message.
The test does not work yet. I think the issue is that we always fail to
upload to s3 store, and we abort the reconcile early and never reach the
code checking the VR condition.
Example log from failing test:
```
2024-11-06T20:41:06.742+0200 INFO controllers.VolumeReplicationGroup Entering reconcile loop {"VolumeReplicationGroup": {"name":"vrg-mxapi","namespace":"envtest-ns-mxapi"}, "rid": "95ce7211-0d29-47c1-a588-22ddf6ab0bed"}
2024-11-06T20:41:06.746+0200 INFO controllers.VolumeReplicationGroup Recipe {"VolumeReplicationGroup": {"name":"vrg-mxapi","namespace":"envtest-ns-mxapi"}, "rid": "95ce7211-0d29-47c1-a588-22ddf6ab0bed", "elements": {"PvcSelector":{"LabelSelector":{"matchLabels":{"appclass":"platinum","environment":"dev.AZ1-mxapi"}},"NamespaceNames":["envtest-ns-mxapi"]},"CaptureWorkflow":null,"RecoverWorkflow":null}}
2024-11-06T20:41:06.746+0200 INFO controllers.VolumeReplicationGroup Fetching PersistentVolumeClaims {"VolumeReplicationGroup": {"name":"vrg-mxapi","namespace":"envtest-ns-mxapi"}, "rid": "95ce7211-0d29-47c1-a588-22ddf6ab0bed", "pvcSelector": "appclass=platinum,environment=dev.AZ1-mxapi"}
2024-11-06T20:41:06.746+0200 INFO controllers.VolumeReplicationGroup Found 2 PVCs using label selector appclass=platinum,environment=dev.AZ1-mxapi {"VolumeReplicationGroup": {"name":"vrg-mxapi","namespace":"envtest-ns-mxapi"}, "rid": "95ce7211-0d29-47c1-a588-22ddf6ab0bed"}
2024-11-06T20:41:06.746+0200 INFO controllers.VolumeReplicationGroup Found PersistentVolumeClaims {"VolumeReplicationGroup": {"name":"vrg-mxapi","namespace":"envtest-ns-mxapi"}, "rid": "95ce7211-0d29-47c1-a588-22ddf6ab0bed", "count": 2}
2024-11-06T20:41:06.748+0200 ERROR controllers.VolumeReplicationGroup.vrginstance Kube object protection store inaccessible {"VolumeReplicationGroup": {"name":"vrg-mxapi","namespace":"envtest-ns-mxapi"}, "rid": "95ce7211-0d29-47c1-a588-22ddf6ab0bed", "State": "primary", "name": "s3profile3", "error": "bucket 'bucketFail2' invalid"}
github.com/ramendr/ramen/internal/controller.s3StoreAccessorsGet
/Users/nsoffer/src/ramen/internal/controller/s3_store_accessors.go:34
github.com/ramendr/ramen/internal/controller.(*VRGInstance).s3StoreAccessorsGet
/Users/nsoffer/src/ramen/internal/controller/volumereplicationgroup_controller.go:1458
github.com/ramendr/ramen/internal/controller.(*VRGInstance).processVRG
/Users/nsoffer/src/ramen/internal/controller/volumereplicationgroup_controller.go:562
github.com/ramendr/ramen/internal/controller.(*VolumeReplicationGroupReconciler).Reconcile
/Users/nsoffer/src/ramen/internal/controller/volumereplicationgroup_controller.go:453
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
/Users/nsoffer/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
/Users/nsoffer/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
/Users/nsoffer/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
/Users/nsoffer/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:224
2024-11-06T20:41:06.748+0200 INFO controllers.VolumeReplicationGroup.vrginstance Entering processing VolumeReplicationGroup for deletion {"VolumeReplicationGroup": {"name":"vrg-mxapi","namespace":"envtest-ns-mxapi"}, "rid": "95ce7211-0d29-47c1-a588-22ddf6ab0bed", "State": "primary", "Finalize": true}
2024-11-06T20:41:06.748+0200 INFO controllers.VolumeReplicationGroup.vrginstance pvc does not contain VR protection finalizer. Skipping it {"VolumeReplicationGroup": {"name":"vrg-mxapi","namespace":"envtest-ns-mxapi"}, "rid": "95ce7211-0d29-47c1-a588-22ddf6ab0bed", "State": "primary", "Finalize": true, "pvc": "envtest-ns-mxapi/pvc-mxapi-1"}
2024-11-06T20:41:06.748+0200 INFO controllers.VolumeReplicationGroup.vrginstance pvc does not contain VR protection finalizer. Skipping it {"VolumeReplicationGroup": {"name":"vrg-mxapi","namespace":"envtest-ns-mxapi"}, "rid": "95ce7211-0d29-47c1-a588-22ddf6ab0bed", "State": "primary", "Finalize": true, "pvc": "envtest-ns-mxapi/pvc-mxapi-0"}
2024-11-06T20:41:06.748+0200 INFO controllers.VolumeReplicationGroup.vrginstance Kube object protection {"VolumeReplicationGroup": {"name":"vrg-mxapi","namespace":"envtest-ns-mxapi"}, "rid": "95ce7211-0d29-47c1-a588-22ddf6ab0bed", "State": "primary", "Finalize": true, "disabled": true, "VRG": true, "configMap": false, "for": "deletion"}
2024-11-06T20:41:06.748+0200 INFO controllers.VolumeReplicationGroup.vrginstance Delete cluster data in {"VolumeReplicationGroup": {"name":"vrg-mxapi","namespace":"envtest-ns-mxapi"}, "rid": "95ce7211-0d29-47c1-a588-22ddf6ab0bed", "State": "primary", "Finalize": true, "s3Profiles": ["s3profile3"]}
2024-11-06T20:41:06.749+0200 INFO controllers.VolumeReplicationGroup.vrginstance Requeuing due to failure in deleting cluster data from S3 stores {"VolumeReplicationGroup": {"name":"vrg-mxapi","namespace":"envtest-ns-mxapi"}, "rid": "95ce7211-0d29-47c1-a588-22ddf6ab0bed", "State": "primary", "Finalize": true, "errorValue": "delete objects with key prefix envtest-ns-mxapi/vrg-mxapi/ using profile s3profile3, err failed to get client for s3Profile s3profile3, err bucket 'bucketFail2' invalid"}
2024-11-06T20:41:06.749+0200 INFO controllers.VolumeReplicationGroup.vrginstance Exiting processing VolumeReplicationGroup {"VolumeReplicationGroup": {"name":"vrg-mxapi","namespace":"envtest-ns-mxapi"}, "rid": "95ce7211-0d29-47c1-a588-22ddf6ab0bed", "State": "primary", "Finalize": true}
```
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
|
Merged into #1639 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Try to test message propagation.