When using limited privilege environment (roles) empty metrics endpoint

[SennaSemakula]

What happened:
I ran kube-state-metrics using Roles and RoleBindings following: https://github.com/kubernetes/kube-state-metrics?tab=readme-ov-file#limited-privileges-environment but I did not pass in --namespaces as it should default to all namespaces including the current one. When I navigate to localhost:8080/metrics the endpoint is blank.

What you expected to happen:
When using limited privileges (roles and not clusterroles) it should have metrics on the current namespace. Note this works if you do pass in --namespaces but it should work regardless as the default value monitors all namespaces based on: https://github.com/kubernetes/kube-state-metrics/blob/main/pkg/options/options.go#L146

How to reproduce it (as minimally and precisely as possible):

1. Follow steps but make sure you do not specify --namespaces as it should default to all namespaces by default
2. Navigate to localhost:8080/metrics and you should see no metrics

Anything else we need to know?:
I'm working on deploying kube-state-metrics in a tenanted environment that is using: https://github.com/kubernetes-sigs/hierarchical-namespaces.

Environment:

• kube-state-metrics version: 2.12.0
• Kubernetes version (use kubectl version): v1.27.12-gke.1115000
• Cloud provider or hardware configuration: GKE
• Other info:

[thunderbird86]

Have the same issue, but I didn't setup any limited privileges environment, it works fine for some time and now it has the same empty responce

[Serializator]

The problem originates from the use of v1.NamespaceAll, where KSM assumes it can list and watch resources at the cluster scope. This is a wrong assumption in a limited privilege environment.

W0623 13:07:43.643600  104637 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:kube-state-metrics" cannot list resource "pods" in API group "" at the cluster scope

https://github.com/kubernetes/kube-state-metrics/blob/main/internal/store/builder.go#L514-L524

The thing is though, without the permission to list namespaces, KSM can't determine what namespaces we have access to either. The minimal requirement in a limited privilege environment for something like this to work is a cluster role (binding) to allow the listing of namespaces across the cluster.

[dgrisonnet]

/assign @mrueg
/triage accepted