Remove hard-coded namespace from Helm Chart 'projectcalico/tigera-operator'

[davivcgarcia]

Expected Behavior

When installing the Helm Chart, the user expects that namespace is configurable by flag --namespace. However the current version fixes namespace and doesn't respect the convention.

https://helm.sh/docs/faq/changes_since_helm2/#release-names-are-now-scoped-to-the-namespace

Current Behavior

If you render the current chart version, you'll see that many resources are created with hard-coded namespace tigera-operator.

helm template calico projectcalico/tigera-operator --namespace dummy

Possible Solution

Update the Helm Chart to respect the namespace definition provided by the user when supplied --namespace, and recommend the usage of tigera-operator in the docs. Also, drop the creation of the namespace and recommend the usage of the flag --create-namespace instead.

Context

The way the chart is written today is breaking the GitOps workflow using FluxCD.

[davivcgarcia]

This issue was initially raised at tigera/operator#1434 and tigera/operator#1427.

[stevehipwell]

I've got an alternative chart that supports running the operator in a custom namespace.

[ArchiFleKs]

Right now it is not possible to install the operator inside the namespace tigera-operator which is annoying because the chart itself tries to create it.

[caseydavenport]

@davivcgarcia marked this as a bug. I am curious though why the tigera-operator namespace isn't suitable / why you need to change it?

[caseydavenport]

Actually swapping this to enhancement - I think it's working as intended right now, but that doens't mean we can't change it. I do need to understand the motivation for wanting to install in a separate namespace though.

[ArchiFleKs]

@caseydavenport the namespace is suitable but the helm release cannot be install in the tigera operator namespace because it already exist (either created by helm or by another tools before (in my case wit terraform for example). The current state of thing is :

• deploy into « tigera/dummy » namespace
• The helm release create the tigera-operator namespace but the helm v3 release does not reside inside the tigera operator namespace. Which is counter intuitive.

I think the namespace should juste be removed and left to the user or at least not created via Helm.

Hope it is clearer. Helm v3 release need to be install in a namespace, which is specify when running Helm install/upgrade command. This namespace need to exist or can be created by helm with « —create-namespace » but this chart create the namespace also.

TLDR: Helm install tigera-operator -n tigera-operator does not work.

Edit: If you need people to use the tigera operator namespace that’s fine if it is specified in the docs or elsewhere but people need to be able to decide how the namespace creation will occurred without having no choice that being created and managed by the Helm release

[caseydavenport]

the namespace is suitable but the helm release cannot be install in the tigera operator namespace because it already exist (either created by helm or by another tools before (in my case wit terraform for example)

If I understand correctly, you mean that there is already a deployment of the tigera-operator on the cluster?

If that's the case, you definitely shouldn't be installing a second one in a new namespace. That's a surefire way to break your cluster. Sorry if I'm still misunderstanding, though!

[ArchiFleKs]

No what I mean is that helm need a namespace to store the release, in the sense of storing the helm release in a secret. With this chart, there are 3 use case:

• you do not create the namespace before hand and the chart fail to install if you run: helm install -n tigera-operator tigers-operator because the namespace does not exist

• you run helm with create namespace options and then the chart fails to install because the namespace tigera-operator already exist (because it was created before by Helm) and then the template in the chart tries to create it also but it already exist

• you install the chart in another namespace like « dummy » and it works so in the end you get 3 namespaces :

  • dummy: where the helm release secret is stored
  • Tigera-operator where the operator pod reside
  • Calico-system where the calico pods reside

@caseydavenport did you tried to helm install in the namespace tigera-operator, like helm install -n tigera-operator $CHART_PATH ?

[ArchiFleKs]

the namespace is suitable but the helm release cannot be install in the tigera operator namespace because it already exist (either created by helm or by another tools before (in my case wit terraform for example.

By that I mean namespace are kind of privileged resources and people might want to manage them in a different way. Almost no chart embed a namespace object. But Helm offer the option to create it or not.

For example when I managed cluster with Terraform I’m creating the namespace with terraform and not letting Helm automatically create the namespace, so we can add annotation or labels etc. Then I install the release in the terraform created namespace.

[ArchiFleKs]

Here is the outputs on a fresh minikube cluster:

 helm install -n tigera-operator calico projectcalico/tigera-operator --version v3.20.0 
Error: create: failed to create: namespaces "tigera-operator" not found

Then if a create the namespace with Helm:

helm install -n tigera-operator calico projectcalico/tigera-operator --version v3.20.0 --create-namespace      
Error: namespaces "tigera-operator" already exists

The official doc install into the default namespace:

 helm install calico projectcalico/tigera-operator --version v3.20.0                                          ░▒▓ INT ✘  0.31.3 terragrunt  minikube ⎈  01:44:36   ▓▒░
W0817 01:44:48.216026 1781290 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0817 01:44:48.275522 1781290 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME: calico
LAST DEPLOYED: Tue Aug 17 01:44:48 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

Helm release is installed in default namespace

helm ls -n default
NAME    NAMESPACE       REVISION        UPDATED                                         STATUS          CHART                   APP VERSION
calico  default         1               2021-08-17 01:44:48.051033191 +0200 CEST        deployed        tigera-operator-v3.20.0 v3.20.0

But the pods are in tigera-operator but not the Helm release:

k -n tigera-operator get pods 
NAME                               READY   STATUS    RESTARTS   AGE
tigera-operator-698876cbb5-nt8bz   1/1     Running   0          68s

[caseydavenport]

@ArchiFleKs thanks for the detailed explanation, I think that makes more sense to me now!

Here's what I'm thinking - we need to allow specification of the -n argument to helm so that it can properly put its releases in the tigera-operator namespace. However, we really don't want two versions of the operator installed on the same cluster, so we probably want to make sure that the Deployment, etc is hard-coded to tigera-operator independent of what the --namespace argument is to helm.

I think that means we should:

• Remove the tigera-operator namespace from the helm chart, and move it to be created prior to helm install
• Keep explicit namespaces set on resources within the chart.

One thing I don't fully understand is the implication for upgrades for anyone who is using the existing chart. Would helm delete the namespace on upgrade?

CC @ozdanborne @lwr20

[stevehipwell]

@caseydavenport I don't think you should be using hard coded namespaces to block a second operator being installed in cluster, that's not what Helm is designed for. An idiomatic Helm chart should be able to be installed in any namespace. I get the need for the calico-system namespace but there is no reason why the operator needs to be in the tigera-operator namespace. If you really need to guarantee there is only ever one operator it would need to be a collaborative solution such as a fixed named secret in the installation namespace from the chart to stop multiple installations to the same namespace; combined with the operator maintaining a fixed named secret in calico-system to define the active operator. However after this you're into the 80/20 realm of diminishing returns and would need to think of the effort is worth it for an edge case actively documented against.

RE the Helm chart I'd suggest a v2 release to fix the current shortcomings. I have my own version of the chart which can be installed into any namespace that I'd be happy to offer as a PR.