Introduce automation for testing new rukpak releases

[timflannagan]

Goal: introduce a new GHA workflow that's responsible for running e2e tests against new rukpak releases.

AC:

• Create a kind cluster
• Mirror the quickstep steps that are documented in the "Installation" section of a release
• Wait until the rukpak stack is healthy
• Load the e2e testdata images into that kind cluster
• Run the e2e suite against that custom rukpak release

[nsapse]

Should this live in the existing e2e workflow (i.e. here)?

[timflannagan]

I think we can have a dedicated workflow for this.

[nsapse]

Is there an existing action/package you'd recommend for this part Mirror the quickstep steps that are documented in the "Installation" section of a release or is an old fashioned grepping the MD in order?

[timflannagan]

@nsapse I probably could've worded that better: use the commands that are listed in the "Installation" section of https://github.com/operator-framework/rukpak/releases/tag/v0.3.0 as a concrete reference.

[nsapse]

Ah, alright. I assumed you would want to pull the commands from each release every time. Will just use those currently existing.

[nsapse]

Could I get clarification on this step:

Load the e2e testdata images into that kind cluster

Looking at the current e2e job I don't see any special data being loaded before the tests run.

[timflannagan]

tl;dr Just use the make kind-load-bundles command for this functionality.

There's an existing Makefile command that's responsible for building and loading the testdata images into an existing kind cluster:

$ make help
...
build/load:
  build-container  Builds provisioner container image locally
  kind-load-bundles  Load the e2e testdata container images into a kind cluster
  kind-load        Loads the currently constructed image onto the cluster

The e2e Makefile target, which is what CI uses as the entrypoint to the e2e suite, specifies that kind-load-bundles Makefile target as a prerequisite target:

e2e: KIND_CLUSTER_NAME=rukpak-e2e
e2e: build-container kind-cluster kind-load kind-load-bundles run test-e2e ## Run e2e tests against a kind cluster

[joelanford]

I think this is pretty close to what I was getting at in #55

IMO it would be nice if our e2e process basically assumed that the current commit was a release, such that we:

1. Compile all the binaries
2. Build all the images
3. Start kind, load the images, load the e2e bundles
4. Generate the rukpak release manifest (making sure it references the images we just built)
5. Apply the "release" manifest to the kind cluster (is that just kubectl -k now?)
6. Run the e2e tests
7. If everything passes, tag and push the images if we're on main or a tag.

[tylerslaton]

@timflannagan @nsapse @joelanford Would it make sense to add a job into the existing release workflow here that runs the e2e suite? That way every release has a prerequisite of the e2e suite failing.

[joelanford]

@tylerslaton that's pretty much what I had in mind. Unfortunately, the make release step of that workflow that does steps 1,2 and 7 above. So we'd need to break things up a bit to do that.

Another consideration: by the time we're running that workflow on the main branch or on a tag, there isn't really much point in running the e2e because we've already pushed the thing we want to release. In that case, running the e2e is just an opportunity for the e2e to flake and mess up the release/push for that branch/tag.

So my suggestion would be:

1. For PRs, do what I said in my previous comment
2. For main/tags, skip steps 3-6.