ClusterTriggerAuthentication does not work with namespaced authenticationRef

[Amir-Wollman]

Report

Background
I'm trying to implement Keda to scale a deployment that has multiple instances across multiple namespaces. In order to not duplicate the authentication trigger I want to create a cluster trigger authentication which uses a secret to set the auth header and value.

Manifest used

apiVersion: v1
items:
- apiVersion: keda.sh/v1alpha1
  kind: ClusterTriggerAuthentication
  metadata:
    name: keda-prom-creds
  spec:
    secretTargetRef:
    - key: PRIVATE_KEY
      name: secret-keys
      parameter: customAuthValue
    - key: authToken
      name: secret-keys
      parameter: customAuthHeader
kind: List

Expected Behavior

Keda will create a ClusterTriggerAuthentication resource which is usable in all namespaces

Actual Behavior

ClusterTriggerAuthentication looks for the secret in kube-system namespace and is unable to find the secret since it is created in a different namespace and no parameter is accepted in schema to pass namespace.
This causes the below err in scaledObject:

Events:
  Type     Reason                   Age                From           Message
  ----     ------                   ----               ----           -------
  Warning  ScaledObjectCheckFailed  6s (x81 over 20h)  keda-operator  failed to ensure HPA is correctly created for ScaledObject
  Warning  KEDAScalerFailed         6s (x9 over 49m)   keda-operator  error parsing prometheus metadata: no custom auth header given

This behavior was verified by replacing ClusterTriggerAuthentication with TriggerAuthentication as follows:

apiVersion: v1
items:
  - apiVersion: keda.sh/v1alpha1
    kind: TriggerAuthentication
    metadata:
      namespace: keda-tenant
      annotations:
      name: namespaced-keda-prom-creds
    spec:
      secretTargetRef:
        - key: PRIVATE_KEY
          name: secret-keys
          parameter: customAuthValue
        - key: authToken
          name: secret-keys
          parameter: customAuthHeader
kind: List

Which does work.

Steps to Reproduce the Problem

1. Create a ClusterTriggerAuthentication which points at any secret (since this resuorce is namespaced)
2. Use that resource in any namespaced scaledObject

Logs from KEDA operator

2023-08-22T10:05:32Z    ERROR   scale_handler   error trying to get secret from namespace       {"type": "ScaledObject", "namespace": "keda-tenant", "name": "prometheus-so-test-metric", "Secret.Namespace": "kube-system", "Secret.Name": "secret-keys", "error": "Secret \"secret-keys\" not found"}
github.com/kedacore/keda/v2/pkg/scaling/resolver.resolveAuthSecret
        /workspace/pkg/scaling/resolver/scale_resolvers.go:515
github.com/kedacore/keda/v2/pkg/scaling/resolver.resolveAuthRef
        /workspace/pkg/scaling/resolver/scale_resolvers.go:238
github.com/kedacore/keda/v2/pkg/scaling/resolver.ResolveAuthRefAndPodIdentity
        /workspace/pkg/scaling/resolver/scale_resolvers.go:182
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers.func1
        /workspace/pkg/scaling/scalers_builder.go:68
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
        /workspace/pkg/scaling/scalers_builder.go:78
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
        /workspace/pkg/scaling/scale_handler.go:360
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
        /workspace/pkg/scaling/scale_handler.go:281
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
        /workspace/controllers/keda/hpa.go:209
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
        /workspace/controllers/keda/hpa.go:75
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA
        /workspace/controllers/keda/hpa.go:48
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
        /workspace/controllers/keda/scaledobject_controller.go:394
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
        /workspace/controllers/keda/scaledobject_controller.go:254
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
        /workspace/controllers/keda/scaledobject_controller.go:177
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:314
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226

2023-08-22T10:05:32Z    ERROR   scale_handler   error resolving auth params     {"type": "ScaledObject", "namespace": "keda-tenant", "name": "prometheus-so-test-metric", "scalerIndex": 0, "error": "error parsing prometheus metadata: no custo
m auth header given"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
        /workspace/pkg/scaling/scalers_builder.go:81
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
        /workspace/pkg/scaling/scale_handler.go:360
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
        /workspace/pkg/scaling/scale_handler.go:281
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
        /workspace/controllers/keda/hpa.go:209
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
        /workspace/controllers/keda/hpa.go:75
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA
        /workspace/controllers/keda/hpa.go:48
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
        /workspace/controllers/keda/scaledobject_controller.go:394
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
        /workspace/controllers/keda/scaledobject_controller.go:254
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
        /workspace/controllers/keda/scaledobject_controller.go:177
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:314
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226

KEDA Version

2.11.2

Kubernetes Version

1.27

Platform

Amazon Web Services

Scaler Details

Prometheus scaler

Anything else?

No response

[Amir-Wollman]

seems related (but not duplicate of):
#4857

edit
after rading the case more carefully, it is indeed unrelated as JorTurFer suggested.
Keeping here though for context :)

[JorTurFer]

Hi
It's not related xD
The linked issue it's to support adding a subset of namespaces instead of all or just 1

Could you share the object where you use the ClusterTriggerAuthentication? (ScaledObject|ScaledJob)

I think that the problem here is a mismatch or expectations, ClusterTriggerAuthentication isn't something like a template that can be reused wherever, it's a unique resource shared across the whole cluster. Due to this, we have to read the secret from somewhere, and in this case is the namespace where KEDA is deployed. If you place your secrets in that namespace, can use the ClusterTriggerAutentication in the whole cluster, using the secret placed in the namespace where KEDA is deployed.
We could support reading secrets from everywhere as part of ClusterTriggerAutentication, but it could be a security breach because you can discover arbitrary secrets blocked for you using KEDA's ClusterTriggerAutentication as the tool for accessing them.

[Amir-Wollman]

• Irelevant
  @JorTurFer true, but implementing a fix for it would require implementing namespace in SecretTargetRef which is also used (as is) in ClusterTriggerAuthentication.

---

I think there's some confusion as for the issue.

I have a secret named secret-keys in monitor namespace, which I want to use in the ClusterTriggerAuthentication, and than reference it in scaledObjects across namespaces. To my understanding, this is the use case for the ClusterTriggerAuthentication.
However, when I create it, Keda by default searches for the secret in kube-system ns as it appears in the log.

edit
I do see your point though with the security breach, still I'd rather not duplicate a secret if it is not a must.
I would say though that these can be scoped to specific namespaces\secrets at the RBAC level to eliminate the security risk

[Amir-Wollman]

I'll duplicate the secret in the keda namespace as a workaround, but if this could be implemented in any sort of way it would be great.

@JorTurFer should I change this in to a feature request instead of a bug?

[JorTurFer]

I don't think that is something to do due to the security implications that it has as commented above. It can be used to ignore RBAC and discover secret information. but let's see other opinions
@kedacore/keda-core-contributors , WDYT?

[Amir-Wollman]

I'm going to proceed and close this issue, but hopefully it can be reopened again once relevant. This PR was opened under the wrong premise that Kubernetes RBAC allowed to grant access to specific namespaced resources by name. Seeing now that this is currently not supported - the suggested solution does become a security risk as suggested by @JorTurFer.

Since this limitation is unlikely to change, as described in this PR and this PR I see no reason to keep this open.

Thank you for your help in reviewing this request :)