Mount our persistent PVC as RWX

[TomasTomecek]

Followup to #105, it contains a lot of context for this work

TL;DR we can only have a single PVC, it is now RWO (only a single pod can have a PV), fedora-infra team will not give us more PVs so we need to use that PVC as RWX: multiple pods can mount it

Tasks:

• prepare a change for our persistent PVC definition to be RWX https://github.com/fedora-copr/log-detective-website/blob/main/openshift/log-detective.yaml#L50
  • we have another namespace in the cluster, you may want to experiment with it there
• once ready, make an announcement in our "log-detective" channel that there will be an outage for one hour
• log in to our cluster and back everything up ( = copy to your laptop) in /persistent - the volume
• take a deep breath and delete the persistent PVC with RWO
• oc apply the log-detective.yaml with the RWX change
• copy all the content inside
  • ⚠️ this will be really tricky because gunicorn requires certificates present in the volume and since the volume will be empty, you won't be able to run a pod; the easiest thing to do is to replace the command with sleep infinity or something like that
• verify all works well, ideally ask people on the team for help :)

[nikromen]

copy all the content inside

I want to empathize here copy the whole /persistent directory (e.g. to your device)