Helm uninstall leaves px-operator namespace in (stuck) terminating state

[ddelnano]

Describe the bug
The helm install process followed by a helm uninstall does not fully clean up all pixie resources in the v0.1.7 operator release. The OLM project added a csv-cleanup finalizer in v0.27.0 that causes the px-operator to get stuck in a terminating state.

To Reproduce
Steps to reproduce the behavior:

1. Install pixie via the helm install instructions
2. helm -n pl uninstall pixie
3. Notice that the px-operator is stuck in a terminating state due to the operators.coreos.com/csv-cleanup finalizer

Expected behavior
Helm uninstall cleans up the olm, px-operator and pl namespaces.

Logs
I don't have the logs handy, but I will attach them to this issue when I trigger the problem again.

[ddelnano]

As a short term workaround, if OLM is installed prior to using the Pixie helm chart, the issue does not occur. This is because OLM won't be a part of the helm installation, which allows the px-operator finalizers defined within OLM to run successfully on helm uninstall.

I'm evaluating options for addressing this finalizer issue in the meantime.

[danlhennessy]

Can confirm I am experiencing the same issue. Have also found that installing via helm, and when uninstalling, running:
px delete
against the specific cluster before helm uninstall has also worked without finalizers causing operator namespace to get stuck. But yes not ideal, preferable for helm to manage everything

[ddelnano]

I have a proof of concept on this branch that implements a potential fix.

That change adds an additional k8s Job that will delete the olm operator namespace (px-operator by default) on the helm pre-delete hook. This allows olm to stay running before helm starts deleting resources and ensures the csv-finalizer has time to run.

I'll be working to get a PR open for this once I get feedback on this approach and test out an operator rc build.

[ddelnano]

#2059 is nearing completion and I hope to have an operator release out with it in the next week.

[alihamza-official]

#2059 is nearing completion and I hope to have an operator release out with it in the next week.

Thanks @ddelnano for the fix. Looking forward to it.

[ddelnano]

The v0.1.7 release is now available!