error deleting namespace on tilt down

[landism]

A user in #tilt reported this on tilt down:

Error: Deleting k8s entities: kubectl delete:
stderr: Error from server (Conflict): error when deleting "STDIN": Operation cannot be fulfilled on namespaces "foo": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.
: exit status 1

They said the namespace went away a few seconds later. It'd be better if Tilt 1) did not generate this error, and 2) could wait to exit until the namespace was gone, in case the next command relies on it being gone.

I'm not clear on whether this is a tilt-specific bug that tilt can do much about, or if it's just a general k8s thing.

There are lots of hits on Google and k8s slack for "The system is ensuring all content is removed from this namespace", so this definitely seems like a common k8s thing that Tilt isn't making worse. However, all the reports I found seemed to be about the namespace being stuck in "terminating" for longer than a few seconds, and requiring manual intervention to delete, rather than simply waiting.

Most of the descriptions of this issue seem to indicate that it's because k8s is waiting for a finalizer to run. kubectl delete --help says --wait If true, wait for resources to be gone before returning. This waits for finalizers. (default true), which takes some credence from that theory.

I haven't yet tried repro'ing with a slow finalizer.

The user reporting the issue said they suspect it's due to their use of a StatefulSet. I've asked for the output of kubectl api-resources --verbs=list --namespaced -o name \ | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace> immediately after tilt down to try to confirm (from here)

[apolegoshko]

I'm the author of this #message #bug. This is definitely a sequence error, when resource (in my case statefulSet) didn't completely removed from namespace before executing the namespace deletion.

[nicks]

For what it's worth, I've wondered if we should change tilt down to not delete namespaces at all. That's honestly the behavior I usually prefer when using Tilt day-to-day (see: #2805 (comment)), and would make this issue moot.

[nicks]

fixed by #3165