audit followup: k8s-infra GCP projects shouldn't have random services enabled

[spiffxp]

This is a followup to #1675 which was focused just on k8s-staging-* GCP projects

While I haven't found docs for it, I am told that over-privileged users (e.g. organization admins) who use the UI are capable of auto-enabling services as they click from project-to-project or service-to-service. I think I often run into this when I'm using a service in my personal project, and then switch to a kubernetes.io project.

#1859 adds an ensure_only_services "${project}" "${services[@]}" function to auto-disable services that were directly specified, or direct dependencies of what was specified. I wasn't comfortable enabling it by default for everything, and I didn't want to rewrite ensure-prod-storage.sh just yet.

This issue is intended to cover:

• determining what set of services we think should be enabled by default for a given class/type of project
• determining how to add in special-case services for a given project (e.g. prod-only, test-only)
• pruning services until we're sure we have only what we want enabled
• enabling some mechanism to enforce continually, e.g.
  • remove obnoxiously long env var gate, do it every time ensure_project is run by any human running a script
  • setup a custom script that only prunes services, run a prowjob that does this
  • etc.

[spiffxp]

/area infra/auditing

[spiffxp]

As part of #1890 I'm looking at k8s-conform's services. This project should be storage-only, so definitely going to disable compute

k8s-conform:
  intent:
    - secretmanager.googleapis.com
    - storage-api.googleapis.com
    - storage-component.googleapis.com
  enabled:
    - compute.googleapis.com
    - oslogin.googleapis.com
    - secretmanager.googleapis.com
    - storage-api.googleapis.com
    - storage-component.googleapis.com
  expected:
    - secretmanager.googleapis.com
    - storage-api.googleapis.com
    - storage-component.googleapis.com
  to_enable: []
  to_disable:
    - compute.googleapis.com
    - oslogin.googleapis.com

[spiffxp]

#1890 (comment) removed extraneous services for k8s-conform

[spiffxp]

I was looking into whether terraform could make this any easier... and it does not. They used to have a google_project_services resource, but ended up deprecating and deleting it, ref: hashicorp/terraform-provider-google#4587

I'm iterating on an attempt to make ensure_only_services work a little better over in #2016 to close out #1963

[spiffxp]

/kind cleanup
/priority important-longterm
/assign
I'll self assign since I'm noodling. But I could use some help, especially suggestions or prior art.

Ultimately I think the future here is a golang tool or something akin to crossplane rather than the ludicrous bash/jq I have going on

[ameukam]

Ultimately I think the future here is a golang tool or something akin to crossplane rather than the ludicrous bash/jq I have going on

I started to take a look at CRDs provided by config-connector : https://cloud.google.com/config-connector/docs/concepts/resources.
Planned to do a parallel to #1885.

[spiffxp]

From a freshly provisioned project (ref: #2195 (comment))

The existing bash/jq is not walking deps enough. None of the services should be unexpected after this first call.

    plan to enable/disable the following services
    to_enable:
      - compute.googleapis.com
      - containerregistry.googleapis.com
      - logging.googleapis.com
      - monitoring.googleapis.com
      - storage-component.googleapis.com
    to_disable: []
    Operation "operations/acf.p2-783965148085-e3561dac-7684-4ec5-8a4e-087848353038" finished successfully.
    Operation "operations/acf.p2-783965148085-73b68eda-a5fc-45e5-a827-90df93bfa3ef" finished successfully.
    Operation "operations/acf.p2-783965148085-69a4fc7c-d39b-44da-9978-eb05e6bd4abc" finished successfully.
    Operation "operations/acf.p2-783965148085-17922383-9f5a-4e41-b070-fb6d703dc45f" finished successfully.
    Operation "operations/acf.p2-783965148085-de4f0675-e9d1-4b66-8fd6-9ecf662d56cd" finished successfully.
    "INFO: dry-run mode, would run:" gcloud services disable --force oslogin.googleapis.com --project=k8s-infra-e2e-scale-5k-project
    "INFO: dry-run mode, would run:" gcloud services disable --force pubsub.googleapis.com --project=k8s-infra-e2e-scale-5k-project
    "INFO: dry-run mode, would run:" gcloud services disable --force storage-api.googleapis.com --project=k8s-infra-e2e-scale-5k-project
    WARN: ensure_only_services: after enable/disable cycle, still projects to enable/disable: k8s-infra-e2e-scale-5k-project
    intent:
      - compute.googleapis.com
      - containerregistry.googleapis.com
      - logging.googleapis.com
      - monitoring.googleapis.com
      - storage-component.googleapis.com
    enabled:
      - compute.googleapis.com
      - containerregistry.googleapis.com
      - logging.googleapis.com
      - monitoring.googleapis.com
      - oslogin.googleapis.com
      - pubsub.googleapis.com
      - storage-api.googleapis.com
      - storage-component.googleapis.com
    expected:
      - compute.googleapis.com
      - containerregistry.googleapis.com
      - logging.googleapis.com
      - monitoring.googleapis.com
      - storage-component.googleapis.com
    to_enable: []
    to_disable:
      - oslogin.googleapis.com
      - pubsub.googleapis.com
      - storage-api.googleapis.com

I have a much gnarler set of jq lurking somewhere, I've been afraid to PR it since maybe it takes us too far in an unmaintainable direction. But I'm pretty sure it handles this correctly. I'll see if I can dig it up.

[spiffxp]

So I don't know when this happened, but it appears that gcloud services list no longer includes dependency information.

That's gonna put a real damper on automating an audit of this.

[spiffxp]

/milestone v1.23
I would like to at least get a picture of how random the enabled services are

[ameukam]

/milestone v1.24

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ameukam]

/remove-lifecycle stale
/lifecycle frozen
/milestone clear