Enable secret rotation on Prow without sharing it in plain text

[chewong]

Sub-issue of #1637.

We have an Azure subscription dedicated to testing for various Kubernetes-related projects. It's been running well until we have to rotate this specific secret, which includes sharing with the test-infra on-call folks our credential (i.e. username, password, subscription ID, storage account key, etc all in plain text) used to authenticate test instances to create Kubernetes clusters on our Azure subscription. It's definitely not ideal and we would like to eliminate this process.

We would like to explore different ways to perform secret rotation without sharing it in plain text. Based on the wg-k8s-infra meeting on Feb 17, 2021, we discussed moving Azure ProwJobs from the default Prow build cluster (owned by Google) to a community build cluster and setting up IAM roles so that core contributors from Azure can perform secret rotation by ourselves.

/cc @spiffxp

[spiffxp]

/kind feature
/wg k8s-infra
/sig testing
/priority important-longterm
/area access
/area prow

There are a few parts here:

• establish a policy of using google secret manager to handoff secrets to test-infra-oncall (you get write access, k8s-infra-prow-owners@ get full access)
• consider fleshing out TestTrustedJobSecretsRestricted in kubernetes/test-infra to enforce at presubmit time which jobs are allowed to reference which secrets
• consider migrating jobs to community-owned k8s-infra-prow-build cluster
• consider setting up secretmanager->k8s secret syncing for prow build cluster(s) to remove toil of test-infra-oncall manually applying secrets (could use kubermetes-sigs/k8s-gsm-tools, https://github.com/external-secrets/kubernetes-external-secrets#gcp-secret-manager, etc.)

I'm going to work on the first part since I'm working on writing up some kind of prow playbook in general, and may have time to try a sample PR to demonstrate the second part. Setting up secret syncing is going to take more bandwidth than I have to offer for at least the next two weeks, though I'm happy to assist/empower anyone who wants to try

[chewong]

Thanks for the detailed response, really appreciate it.

establish a policy of using google secret manager to handoff secrets to test-infra-oncall (you get write access, k8s-infra-prow-owners@ get full access)

Thanks for leading this. Let me know if there is anything I can help.

consider fleshing out TestTrustedJobSecretsRestricted in kubernetes/test-infra to enforce at presubmit time which jobs are allowed to reference which secrets

We use the label preset-azure-cred to specific the secret instead of mounting the secret in the job config. I think I have to create a new unit test case for Azure-specific jobs.

consider migrating jobs to community-owned k8s-infra-prow-build cluster

I will bake some of our jobs in k8s-infra-prow-build to make sure nothing breaks before moving all of our jobs there.

consider setting up secretmanager->k8s secret syncing for prow build cluster(s) to remove toil of test-infra-oncall manually applying secrets (could use kubermetes-sigs/k8s-gsm-tools, https://github.com/external-secrets/kubernetes-external-secrets#gcp-secret-manager, etc.)

This would be ideal. I will focus on the above three bullet points before jumping into this one.

[spiffxp]

#1731 to cover the first part

[spiffxp]

/assign @spiffxp
for #1731
/assign @chaodaiG
FYI, ignore the "move to community build cluster" part and I think this overlaps with at least part of a proposal you're working on

[chaodaiG]

kubernetes/test-infra#21484 enabled prow cluster secrets synced from google secret manager and other secret manager providers. kubernetes/test-infra#21610 adds more instructions to make it more clear how clients can rotate secrets by themselves

[spiffxp]

/milestone v1.21