Delete unused prometheus volume on cluster down #1863
Conversation
cli/cmd/cluster_gcp.go
Outdated
| } | ||
| _, _, err = runGCPManagerAccessCommand(uninstallCmd, *accessConfig, nil, nil) | ||
| if err != nil { | ||
| exit.Error(err) |
There was a problem hiding this comment.
For cluster down, it is generally preferable to have best-effort deletes.
Consider the following scenario: the first cluster-gcp down request fails on the delete cluster step. In the next retry, the cluster-gcp down command fails at the removing the volumes step because the previous cluster down attempt prevented us from getting kubectl config. The user can't retry cluster-gcp down because it fails at an unrelated step.
| @@ -97,6 +98,7 @@ func clusterInit() { | |||
| addClusterRegionFlag(_clusterDownCmd) | |||
| addAWSCredentialsFlags(_clusterDownCmd) | |||
| _clusterDownCmd.Flags().BoolVarP(&_flagClusterDisallowPrompt, "yes", "y", false, "skip prompts") | |||
| _clusterDownCmd.Flags().BoolVar(&_flagClusterDownKeepVolumes, "keep-volumes", false, "keep cortex provisioned persistent volumes") | |||
There was a problem hiding this comment.
This flag might be renamed to --keep-cloud-resources so that it can be used to preserver volumes, buckets, policies and log groups. For now I think it is fine.
|
|
||
| function uninstall_grafana() { | ||
| kubectl delete statefulset --namespace default grafana >/dev/null | ||
| kubectl delete pvc --namespace default grafana-storage >/dev/null |
There was a problem hiding this comment.
In a scenario where cluster-down fails, it may not be possible to get kube config to delete the volumes via kubectl. We should use cloud specific apis to find and delete volumes without having to rely on kubectl.
checklist:
make testandmake lint