[Feature Request]: Implement least-privilege access for GKE workload hosting Beam metrics. #40214

What would you like to happen?

Goal: Grafana code has minimal permissions in apache-beam-testing it needs to surface Beam metrics, instead of operating under default compute engine service account.

My understanding of the process:

1. We create a new, least-privilege IAM service account in apache-beam-testing (e.g., grafana-sa), We grant this SA the necessary roles to fetch the metrics (e.g., roles/bigquery.dataViewer if needed).

2. We create a Kubernetes Service account (KSA). This is the identity Grafana Pod will eventually use.

3. We create an IAM binding that maps KSA to IAM SA

4. We configure the GKE cluster and its node pools to use Workload Identity

5. We configure the Grafana GKE deployment to use the KSA we created.

I never done this so my undestanding might have flaws. Docs:

https://docs.cloud.google.com/kubernetes-engine/docs/how-to/service-accounts#service-account-comparison
https://docs.cloud.google.com/kubernetes-engine/docs/concepts/workload-identity

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Prism Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner