[Bug] Ingress.extensions "grafana-ingress" is invalid - pathType must be specified on Kubernetes 1.20+

[GusAntoniassi]

Describe the bug
I am not sure whether this is a bug in the opeator behavior, or just the documentation missing a note, but since I haven't found anything related in issues or docs I'm opening this issue.

When running Grafana Operator with kind in Kubernetes version 1.20 or 1.21, I get the following error when deploying using the default Grafana.yaml available in deploy/examples:

2021-07-25T18:53:30.880-0300    DEBUG   controller-runtime.manager.eventsWarning  {"object": {"kind":"Grafana","namespace":"grafana-operator","name":"example-grafana","uid":"ce1178b4-adaf-490c-a982-767c2a3a53be","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"1264"}, "reason": "ProcessingError", "message": "Ingress.extensions \"grafana-ingress\" is invalid: spec.rules[0].http.paths[0].pathType: Required value: pathType must be specified"}

I have been able to fix this issue by adding the following two attributes to Grafana.yaml in spec.ingress:

    path: /
    pathType: Prefix

If I add just the pathType: Prefix field the error message changes to:

2021-07-26T09:12:34.486-0300    DEBUG   controller-runtime.manager.events        Warning {"object": {"kind":"Grafana","namespace":"grafana-operator","name":"example-grafana","uid":"1c0340b9-1e21-4e6b-8e62-920db186e8cf","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"1614"}, "reason": "ProcessingError", "message": "Ingress.extensions \"grafana-ingress\" is invalid: spec.rules[0].http.paths[0].path: Invalid value: \"\": must be an absolute path"}

I think I can submit a PR to fix this issue, but I'd like to know if it's better to just document the required values in the example Grafana.yaml file or change the operator behavior to automatically add a default value for this field.

To Reproduce
Steps to reproduce the behavior:

1. Create kind cluster with the command kind create cluster --image=kindest/node:v1.20.7
2. Deploy the Grafana Operator by running: make install && make cluster/prepare/local
3. Run the Operator locally with make run
4. Ingress object is not created, and the processing error message is seen in the logs

Expected behavior
Operator should automatically add a default value OR the example Grafana.yaml file should be updated to document the required fields in the affected Kubernetes versions.

Suspect component/Location where the bug might be occuring
unknown

Runtime (please complete the following information):

• OS: Linux
• Grafana Operator Version: Latest (running from master branch)
• Environment: Kind version 0.11.1, Kubernetes 1.21.1 and 1.20.7
• Deployment type: running the operator locally

Additional context
I believe this related to a change introduced in Kubernetes v1.19, where the Ingress pathType no longer has a default value, and another change introduced in Kubernetes v1.20, which fixes field validation when the host field contains a wildcard.

I have tested with Kubernetes v1.19.11 and could not reproduce the same problem.

[hubeadmin]

Hey @GusAntoniassi , Thanks for reporting this, i've seen this issue crop up just after installing the enwest version of minikube, how would you feel about sending a PR to add this as our default in code? (It's fine if you don't have the time, we'll fix this before the next release).

For those looking for a solution, As the OP mentioned. adding the following to your grafana.yaml

  ingress:
    enabled: true
    pathType: 'Prefix'
    path: '/'

will provide a temporary solution to get around this on newer versions of Kubernetes

[GusAntoniassi]

Sure thing @HubertStefanski I can send the PR, the fix would be just to update the Grafana.yaml file?

Do you think it might break compatibility for those running the operator on v1.16 or older? (I can test that with Kind, but asking just in case you already know how the operator would behave)

Edit: I just tested here and got the following error in v1.16.15 and v1.18.19, regardless of setting pathType or not:

2021-07-26T20:23:49.807-0300    ERROR   grafana-controller      error reading state     {"error": "no matches for kind \"Ingress\" in version \"networking.k8s.io/v1\""}
github.com/go-logr/zapr.(*zapLogger).Error
        /home/gus/.asdf/installs/golang/1.16.3/packages/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/log.(*DelegatingLogger).Error
        /home/gus/.asdf/installs/golang/1.16.3/packages/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/log/deleg.go:144
github.com/integr8ly/grafana-operator/controllers/grafana.(*ReconcileGrafana).Reconcile
        /home/gus/workspace/opensource/grafana-operator/controllers/grafana/grafana_controller.go:174
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /home/gus/.asdf/installs/golang/1.16.3/packages/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /home/gus/.asdf/installs/golang/1.16.3/packages/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
        /home/gus/.asdf/installs/golang/1.16.3/packages/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
        /home/gus/.asdf/installs/golang/1.16.3/packages/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
        /home/gus/.asdf/installs/golang/1.16.3/packages/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
        /home/gus/.asdf/installs/golang/1.16.3/packages/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /home/gus/.asdf/installs/golang/1.16.3/packages/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
        /home/gus/.asdf/installs/golang/1.16.3/packages/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
        /home/gus/.asdf/installs/golang/1.16.3/packages/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
2021-07-26T20:23:49.807-0300    DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"grafana-operator","name":"example-grafana","uid":"9d13a210-60a6-4a14-b0b6-e76ad8079928","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"620"}, "reason": "ProcessingError", "message": "no matches for kind \"Ingress\" in version \"networking.k8s.io/v1\""}

I understand they are already EOL, but just to confirm are they not supported anymore by the operator?

[pb82]

@GusAntoniassi the Grafana.yaml in the operator repository is only meant as an example and doesn't set a path type.

Regarding the error above, this sounds like you're running v3 of the operator against Kubernetes 1.20+. You might want to try with 4.0.0 once that is released. The problem should be fixed there.

[GusAntoniassi]

Hey @pb82 from what I could gather this option is mandatory for ingresses in v1.20+ so I think it would be nice to add in the example to avoid some head scratching when testing out the operator or contributing to the project.

Regarding the no matches for kind \"Ingress\" in version \"networking.k8s.io/v1\" error mentioned above, It appeared when running the operator on master branch, against a Kubernetes cluster (created with Kind) on versions 1.16.15 and 1.18.19.

The v4 release will be based off the master branch or is there another branch I could test it?

PS: I created pull request #490 in case it is useful to update the example Grafana.yaml file

[pb82]

@GusAntoniassi thanks for the PR, it has been merged now. The 4.0.0 release is based off the master branch, so you can always test it from there.
I'm wondering if we should also set a default path (/) when creating an ingress and no explicit path is set by the user?

[pb82]

@vmanley this might be something for you

[vmanley]

@pb82 Will have a look. Thanks