chore(e2e): naively parallelize CI jobs by chunking alphabetically

[estroz]

Description of the change: parallelized e2e tests (for upstream CI only)

Motivation for the change: current sequential runs are ~1h10m. the longest parallel run is ~24 min, and could be lower if we increase the matrix size. you will hit Amdahl's law quickly from test setup. it's a little hacky but adjusts automatically when new specs are added/removed.

I'll add some tests next week

Reviewer Checklist

• Implementation matches the proposed design, or proposal is updated to match implementation
• Sufficient unit test coverage
• Sufficient end-to-end test coverage
• Docs updated or added to /doc
• Commit messages sensible and descriptive

[estroz]

This was tested on my private fork, see https://github.com/estroz/operator-lifecycle-manager/actions/runs/1565786458

[timflannagan]

This is really, really (albeit a bit hacky) cool work man - this will infinitely improve the current state of long, frustrating e2e feedback. I just had a couple of questions, and I'd like to land anything that substantially improves the e2e feedback, but we'll need to update the branch protection for our tide configuration (or remove tide entirely which I think is my preference).

[estroz]

@timflannagan the main things I want to add before this merges is tests for the script, and some deduplication by prefix (if both Subscriptions and Subscripts blah blah exist, the latter will be run twice if they are in different chunks). I'll be working on that this week so this can get in asap.

[timflannagan]

I think the overall changes are reasonable to me looking back at that this but my brain is pretty much mush at this point. I'd like to land these changes ASAP but we'd also need to land a tide configuration update as well. I should be back to full capacity by tomorrow.

[timflannagan]

Any particular reason you settled on a trie implementation vs. a more primitive implementation of sorting test specs and chunking arbitrarily?

[estroz]

Primarily to prevent one spec being run by two different chunks. Imagine running two specs defined by phrases "Foo" and "Foo bar" naively in two chunks: the first chunk (^Foo.+) would run both specs, while the second chunk (^Foo bar.+) would run only the latter spec. By using a word trie, the phrases are reduced down to just the word "Foo", which will run both specs in one chunk. At first the downside appears to be a less even distribution of chunk runtimes, since "Foo" and "Foo bar" will always be run together; however this downside also affects naive chunking because spec phrase prefixes are still shared.

[perdasilva]

This is probably a really naive question, but, why not just use Ginkgo's parallelization: e.g. ginkgo -procs=N?

[estroz]

-procs gives you core parallelism, the actions matrix + chunks gives you core and runner parallelism. Runner parallelism is important here because we cannot choose machine size.

[timflannagan]

It's also important to note that the way our CI environment is setup, we create a kind cluster everytime we spawn a new e2e process, so this PR simply builds on top of that existing setup and chunks test specs and feeds those test specs as focus regex groups that an individual kind cluster will then run. There's likely a cleaner solution going forward, but we'd have to shift around some of our current setup to get it working correctly.

[timflannagan]

I think I'm comfortable enough with the implementation to avoid holding up this PR with any further comments given it helps improve QoL quite a bit. I would like to see us open an issue around aggregating the debug/junit/etc. artifacts into a single source, but that can be done asynchronously anyways.

/lgtm

[timflannagan]

Interesting, the extra info-level logging is showing up as errors in the run output:

Run make e2e-local E2E_TEST_CHUNK=1 E2E_TEST_NUM_CHUNKS=4 E2E_NODES=2 ARTIFACTS_DIR=./artifacts/
Error: 2022/01/05 20:31:52 test/e2e/e2e_test.go: found no top level describes, skipping
Error: 2022/01/05 20:31:52 test/e2e/like_metric_matcher_test.go: found no top level describes, skipping
Error: 2022/01/05 20:31:52 test/e2e/setup_bare_test.go: found no top level describes, skipping
...

[timflannagan]

/approve
/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: estroz, timflannagan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [timflannagan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment