Allow `kapp delete` even if default namespace was part of the application configuration

[ciberkleid]

Describe the problem/challenge you have
If the default namespace is included as a resource in a kapp-deployed application, you can then not delete the application. You get the following error and you are stuck:

Error: resource deletion failed: kapp: Error: Applying delete namespace/default (v1) cluster:
  Deleting resource namespace/default (v1) cluster: namespaces "default" is forbidden: this namespace may not be deleted (reason: Forbidden)
. Delete failed: Error (see .status.usefulErrorMessage for details)
Error: exit status 1

Describe the solution you'd like
I don't think it is realistic to prevent the default namespace to be part of an application configuration, so maybe one of these two approaches could work:

• If default namespace is one of the resources in the YAML, ignore it but deploy the rest of the resources in the application
• If default namespace is one of the resources in the kapp-deployed application, do not try to delete the namespace during kapp delete.... Simply ignore it and delete the rest of the application resources.

Anything else you would like to add:
Here is one case in which I ran into this problem:

vmware-tanzu/community-edition#3067

It is not the only situation in which I've experienced this issue, but I think it's a good example.

[100mik]

Adding steps to reproduce this issue using kctrl

1. Add the TCE repo

kctrl package repo add -r tce --url projects.registry.vmware.com/tce/main:0.10.0 

2. Install cert-amanger into default ns

kctrl package install -i cert-man -p cert-manager.community.tanzu.vmware.com --version 1.6.1 --values-file - << EOF                                                                                                                  
---
namespace: default
EOF

3. Attempt a delete

kctrl package installed delete -i cert-man                                                                                                                                                                                           
Delete package install 'cert-man' from namespace 'default'

Continue? [yN]: y

Target cluster 'https://127.0.0.1:49961' (nodes: minikube)

Deleting package install 'cert-man' from namespace 'default'

Waiting for deletion of package install 'cert-man' from namespace 'default'

10:51:37AM: packageinstall/cert-man (packaging.carvel.dev/v1alpha1) namespace: default: Deleting
10:51:43AM: packageinstall/cert-man (packaging.carvel.dev/v1alpha1) namespace: default: DeleteFailed

kctrl: Error: packageinstall/cert-man (packaging.carvel.dev/v1alpha1) namespace: default:
  Deleting: packageinstall/cert-man (packaging.carvel.dev/v1alpha1) namespace: default:
    Deleting: kapp:
      Error:
        Applying delete namespace/default (v1) cluster:
  Deleting resource namespace/default (v1) cluster:
    API server says: namespaces "default" is forbidden: this namespace may not be deleted (reason: Forbidden)
. Delete failed: Error (see .status.usefulErrorMessage for details)

[100mik]

There are a few ways forward from here,

1. We could make it so that kapp apps do not adopt namespaces that are already created as a guardrail against unintended deletions and that will help in situations like these. The downside is that some users migrating existing resources into kapp applications might want to adopt namespaces in which case a flag toggling this behaviour might become handy. (This is a more generic solution that helps multiple use cases).
   One possible downside I can think of is that the ns won't show up when the app is inspected if not adopted.
2. Since, this situation is especially bothersome when it comes to packages that could possibly include a namespace resource that is overlayed depending upon configuration while the user is installing the application. This could warrant a change in kapp-controller where we filter out resources like the default namespace while deleting/deploying the application using flags exposed by kapp.
3. Packages which allow overlaying of namespace resources should have a guardrail against this while templating the manifests. However, if we do publish this suggestion, packages might take a while to incorporate the same.
4. kapp could have a set of resources, that it cannot operate upon (such as resources which the cluster does not allow deletion for). This however is pretty specific and might not be the best course of action.

There has been some discussion in the past for kapp to ignore resources which cannot be operated on and continue with deployment/deletion of other resources. However, this does not help the packaging use case as the PackageInstall deletion still fails when any of the resources cannot be operated upon.

@joe-kimmel-vmw your input would be super helpful here. How do you feel about option 2? Is it too specific or is it worth consideration.

[joe-kimmel-vmw]

I could be wrong but the most appealing to me was option (1) which keeps the logic in kapp. There's a couple related issues in kc right now:

carvel-dev/kapp-controller#523

and

carvel-dev/kapp-controller#535

[cppforlife]

i do think that kapp should be owner of whatever behaviour we decide on. (kapp-controller doesnt interact with any resource kapp deploys directly).

im not a huge fan of "skipping" deletion of default namespace since it seems like user should never have configured app to include default namespace (though i could see how one may want to apply some labels to default namespace via kapp).

i think with the current feature set the general answer should be that default namespace should be marked as delete-strategy=orphan, but that does mean that configuration authors need to be mindful of default vs non-default namespaces.

[100mik]

Since it is a hard rule that the default namespace cannot be deleted from the cluster. It does make sense for kapp to have a set of such resources hardcoded, for which it uses the "orphan" strategy for deletion.

We should probably add other similar resources to such a list as well.
We will be adding this to our unprioritized backlog and work towards it🚀