Remove flakes from the critical path

[perdasilva]

Description

As a contributor, I don't want my PR to be blocked by flaky tests. I want a way to remove them from the critical path.

Acceptance Criteria

1. List of flaky tests identified
2. Flaky tests no longer block the pipeline
3. If a flaky test fails, contributors have clear instructions or easy tools to demonstrate that their PR does not fail on the flaky tests (e.g. a make target that takes the failing tests, executes them in isolation and produces a report and given the PR number pastes it in as a comment on the PR
4. There is a simple mechanism to mark a test as flaky
5. A plan for dealing with the flaky tests

Suggestion

Let's split the e2e run into two stream: stable (required to pass), flaky (not required). We could run each of the flaky tests one at a time in isolation in parallel with the stable e2e tests. If any of the flaky tests fail, the contributor can use a tool (or make target) to run those same tests in isolation locally. If they still fail, this should signal the author there's a breaking change.