Flaky Test Splitting Ideas and Issues

[tateexon]

Flaky Test Splitting Ideas and Issues
Goals
Run Non-Flaky tests in one job
Run Flaky tests in another job

Build Flags

• Build flags allow us to separate tests into separate jobs
• Issue 1: This affects the whole test file that has the build flag so you can't specify a single test
• Issue 1 Work around: Create a separate test file and move the flaky test into it. Not a great solution

testing.Short

• This allows us to put a small piece of code that will let us skip individual tests based on whether the -short flag is set when running go test
• Issue 1: This allows us to skip the flaky tests but does not leave us with a way to only run the flaky tests
• Issue 2: The devs are already using testing.Short for its intended purpose in several places and this would interfere with that.

Environment Variables

• Doing this we would need to put code into flaky tests to use the env variable to skip the test. This gives us the exact same ability as testing.Short above but also the same first issue.
• Issue 1: This allows us to skip the flaky tests but does not leave us with a way to only run the flaky tests

Custom Skip Function + Test Run Parsing

• We build a custom skip function that formats the skip message for flaky tests.
• We use the format above to parse which tests were flaky skips to build a list of skipped flakes.
• Use list built above to do a separate run that uses the dynamic list of skipped flaky tests.
• The custom function use an environment variable or custom flag to determine whether to skip the test or not. When the variable is not present it will skip the test with the message. When it is present it will run the test. This allows both cases.
• Issues 1: We have to create a custom skip function and convince the devs to use it.

[tateexon]

go test regex does not support look ahead and look behind so we can't just tack Flaky onto test names that we want to be flaky sadly.

[kalverra]

Another solution is to provide a custom flag like -runFlaky bool that the skip function will check for to decide if it should actually run or not, easy way for us to toggle runs.

[tateexon]

Added that option in to the custom skip function part!

[tateexon]

After experimenting more with the test flags it appears that we would need to add the flag to all uses tests TestMain functions for go test to work properly. Putting it in an init in our framework doesn't work sadly and we just get an error saying the flag is provided but not defined because it needed to be set to be part of a main function or a TestMain. It is a bit frustrating.