audit followup: ensure secretmanager secrets and bindings are managed by scripts

[spiffxp]

Caught via #1718 (comment)

I've done the following pattern a few times now (most recently #1696 (comment)):

• someone needs to pass a secret to me, wonders how to transfer it
• I setup a secretmanager secret in the relevant project
  • intended to be k8s secrets in a cluster: the project hosting the cluster
    • kubernetes-public for apps running on aaa
    • k8s-infra-prow-build-trusted for prow jobs or components runnong on prow-build-trusted
    • etc.
• Setup appropriate permissions on the secret (need to figure out a consistent pattern here)
  • Org admins should implicitly get access via their role/owner on the organization
  • Give some sort of "oncall" team ownership of the secret and its versions for break-glass cases
  • Give whomever is handing the secret to us write access to the secret version
• Setup labels on the secret
  • app: foo if its for an app in the foo dir running on aaa
  • group: sig-foo if the app is owned by sig-foo
  • ??? maybe the namespace the secret is destined for?
• Request that for things destined to be k8s secrets, people store an actual secret manifest yaml in the secret

Put it all together and this allows for relatively simple / safe deployment: https://github.com/kubernetes/k8s.io/tree/main/slack-infra#how-to-deploy

Problems with the above:

• it needs to be documented instead of me just happening to kinda/sorta do something consistent
• the secret creation should be scripted

[spiffxp]

/kind cleanup
/kind documentation
/priority important-soon
/area cluster-mgmt
/area access

Labels and areas for relevant sigs:

/sig testing
/area prow
I'd like to set this up as the standard for managing secrets for prow.k8s.io (with possible exception of kubeconfigs that speak to google-internal clusters)

/sig release
/area release-eng
triage-party, publishing-bot, and possibly other secrets

/sig contributor-experience
slack-infra, groups management

/sig scalability
perf-dash

/sig node
node-perf-dash

[spiffxp]

Another piece of followup: I setup a custom secretLister role (ref: #1726) which should allow someone to use the GCP console to manage secrets. It should work such that adding it to a group means they'll be able to list/see secrets, but only those they directly have access to. Need to trial this with someone (maybe @jeefy)

[jeefy]

Happy to help! Just ping. :)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[spiffxp]

/remove-lifecycle rotten
/assign
This might be done, or at the very least the description needs more a refresh. A quick short update:

• there is bash that manages secrets as of infra/gcp: manage more secrets via bash #2078
• we would like to migrate everything to use of kubernetes-external-secrets, ref: Migrate all Opaque kubernetes Secrets on k8s-infra clusters to ExternalSecrets #2220

[spiffxp]

I'd like to add to definition of done that secrets are provisioned the same way for all of our clusters. As of completion of #2220 all prow build cluster secrets are provisioned via terraform. But the secrets for aaa are still handled by bash.

[spiffxp]

Once #3028 merges, all of our secrets will be managed by terraform. What then remains is docs on how to provision a new secret.

The age-old question of "where is the best place for these?" and my best guesses for answers:

• prow: in the respective README's in infra/gcp/terraform/k8s-infra-prow-build and/or somewhere in kubernetes/test-infra
• apps: in "running on community cluster"

[spiffxp]

I'll drop the respective sig labels for each app since they've been migrated and this is more on sig-k8s-infra to doc now

/remove-sig node
/remove-sig release
/remove-sig scalability
/remove-sig testing
/remove-sig contributor-experience

[ameukam]

/milestone v1.24

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ameukam]

/remove-lifecycle stale
/milestone clear
/priority backlog
/lifecycle frozen