Remove Reclaim Storage

[dvaldivia]

Deleting a tenant's storage in Kubernetes is a sensitive and controversial issue. On one hand, some users may want to
delete storage in CI/CD scenarios. On the other hand, if configured in production environments and forgotten, it risks
potential data loss.

Kubernetes does not automatically delete Persistent Volume Claims (PVCs) when a StatefulSet is deleted, due to the risk
involved. This decision is left to the user. The MinIO Operator adheres to this practice as well.

To delete a tenant's storage, you should manually delete the PVCs associated with the tenant. This can be done via
kubectl at the same time you are deleting the tenant. For example, to delete a tenant named tenant in the namespace
ns-1, run the following commands:

kubectl -n ns-1 delete tenant tenant
kubectl -n ns-1 delete pvc -l v1.min.io/tenant=tenant

[harshavardhana]

🍰

[cniackz]

Thank you guys 🙏

[nervo]

Hello, sorry to dig up this old pr, but i would like to add some elements :)

@dvaldivia i totaly understand your point of view when you say that the operator should follow the kubernetes way of letting the end-user to decide to delete PVC(s) of a StatefulSet.

Now, the reclaimStorage was a really useful option in CI/CD scenarios, to handle ephemeral S3 buckets, and remove it leave some users without clean solutions.

Starting from kubernetes 1.23, a new StatefulSet field has emerged: persistentVolumeClaimRetentionPolicy (see: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#persistentvolumeclaim-retention). I might be wrong, but exposing this field in the operator options could lead to a perfect situation: follow kubernetes principles, don't re-invent the wheel, and let the user choose.

What do you think ?

[ramondeklein]

This wouldn't the remove S3 buckets (as you stated in your comment above). It will delete all the PVCs associated with the statefulset (and thus all persistent storage). We could add persistentVolumeClaimRetentionPolicy to the Tenant spec, but we don't want to get into the situation that customers misunderstand this property and will end up deleting all their data when they remove the STS.

If you want to remove your PVCs, then you'll have to do it yourself. We don't want MinIO Operator be the one that deletes your precious data.

[nervo]

@ramondeklein once again, i understand your point of view, and what leads to reclaimStorage, a pure custom/non-standard option, removal :)
The fact that kubernetes now propose a native way to handle the cases where reclaimStorage was useful made me think that could be a nice way. Imho you don't have to justify exposing persistentVolumeClaimRetentionPolicy to avoid data loss prevention, as long as this is a standard and well documented option. This is the operator itself who destroys the PVC's, but kubernetes, in a planned and documented manner, following the options you gave.

(yes, i still try to convince you :) )

[ramondeklein]

I understand your point-of-view, but please understand ours. Our customers use MinIO for their most sensitive data and although -technically- fiddling with these settings is not MinIO's fault, it will be MinIO that will be blamed. Because of that we are very hesitant with automatic deletion of persistent volumes.

If you really need to be able to dispose the object-store, then create an object-store in a separate namespace and trash the namespace. Then you're sure everything is deleted. You may want to create a randomized namespace, so you can run multiple object-stores (in case you run multiple CI/CD pipelines in parallel).

[nervo]

@ramondeklein crystal clear 👍

Sorry to have bothered you, and thx for the tip :)