Streamline k8s deploy process

[themightychris]

Description

Simplifies Kubernetes review and deployment workflow by:

• Removes generating candidate branches
• Removes workflow of opening PRs against release branches to deploy Kubernetes changes
• Generating k8s diffs on any pull requests against main that change relevant paths
• Deploy Kubernetes changes on merge into main

Resolves #3200

How has this been tested?

Manually locally

Additional changes outside PR

• Created new secret: projects/1005246706141/secrets/jupyterhub_jupyterhub-sensitive-helm-values
• Created new secret: projects/1005246706141/secrets/sentry_sentry-sensitive-helm-values
• Deleted unused secret: projects/1005246706141/secrets/sentry-sensitive-yaml
• Upgraded sentry PostgreSQL database from 11 to 15

Post-merge follow-ups

• Recreate sentry-sentry-secret secret from backup

[github-actions]

The following changes will be applied to the production Kubernetes cluster upon merge.

BE AWARE this may not reveal changes that have been manually applied to the cluster getting undone—applying manual changes to the cluster should be avoided.

jupyterhub, hub, Deployment (apps) has changed:
...
          hub.jupyter.org/network-access-proxy-api: "true"
          hub.jupyter.org/network-access-proxy-http: "true"
          hub.jupyter.org/network-access-singleuser: "true"
        annotations:
          checksum/config-map: 0d9ed96aff33a452e22edf6b9b5261b327c0eeba0eefa3fa026b56b6b5b8ffea
-         checksum/secret: d8fa42cf917b35942d883485f40cd762c7119683eb12d57407807398424019e5
+         checksum/secret: 2734177dde342563227a03ecf175c790891b289facd76ca4be33410f3b1d44c3
      spec:
        tolerations:
          - effect: NoSchedule
            key: hub.jupyter.org/dedicated
            operator: Equal
...
jupyterhub, hub, Secret (v1) has changed:
...
    name: hub
  data:
    hub.config.ConfigurableHTTPProxy.auth_token: 'REDACTED # (64 bytes)'
    hub.config.CryptKeeper.keys: 'REDACTED # (64 bytes)'
    hub.config.JupyterHub.cookie_secret: 'REDACTED # (64 bytes)'
-   values.yaml: '-------- # (11050 bytes)'
+   values.yaml: '++++++++ # (10773 bytes)'
  type: Opaque

[SorenSpicknall]

The lint task will go green (or, at the very least, fail for legitimate reasons) once this merges and you bring those changes into this branch.

[SorenSpicknall]

Just one small note on stewardship of a related repo.

I will likely also reopen the associated Issue or create a new documentation-specific Issue after merging to continue some docs changes related to this new workflow. For instance, the /ci README will need some updates. But those don't have to hold up merging these changes.

[SorenSpicknall]

We may need to fork a version of this into Cal-ITP controlled space eventually - probably not actionable right this second, but worth creating a follow-up Issue for.

[themightychris]

Yeah I'm not sure about the best strategy there. This is a repository we actively maintain for many public/community/client consumers. Cal-ITP is pinned to a specific tagged version and it's a public open repo so it's ensured to stay static until someone bumps to a newer version. Eventually dependencies will need to be updated and at that point whoever is maintaining the infra can either fork the template or bump up to a new version. Most likely Jarvus will have already tested and documented the upgrade path to newer dependency mixes.

While there is a goal to minimize Jarvus dependencies, the project is depending on a lot of random public/community GitHub repositories for lots of things like GitHub actions and helm charts and it would be an anti-pattern to go through and fork every dependency. In cases where a Jarvus repository is linked that's just a fork of something else or a single-use repository, I'd agree with prioritizing eliminating that external dependency, but in this case this is a first-class maintained community project so I'm not sure it makes sense as a goal to just fork it into ITP preemptively

[themightychris]

@SorenSpicknall That said it would be a good issue to search for "Jarvus" throughout the codebase and make sure if any forks of community repos are being linked against, we're doing work to get any needed patches moved upstream so we can move the link back to targeting the upstream directly