Dealing with CI / Test Failures / Flakey Tests

[IanMatthewHuff]

Tagging as iteration candidate / triage needed so that we discuss. Quick notes from our Teams meeting on how to help deal with all the flakey test failures that we ran into last iteration.

• Have more distinct areas of code / feature ownership. Would make it clear who needs to deal with a failing test.
• Have a rotating build champ would be in charge of keeping the build green. Investigate anything failing on main CI.
• Outside of endgame don't allow pushing without being all green on the PR. Makes us feel the full pain to deal with it now.
• Better tools for looking at test history, what has been failing on and off or what failed in the most recent main run
• Make sure we tag all flakey test issues and add them to our triage notebook to make sure that they don't get lost. Don't have tests we disable and lose track of.
• For any PRs that have flake failures disable them in that PR and file an issue with the flake tag right then.

[rchiodo]

Proposal:

• Change release plan so that beginning of end game, we make sure all tests are passing. Either disable them or fix them. Release person is responsible for this.
• If during a PR a test is failing, we have an 'owner' for each test. Owner decides if it needs to be fixed by PR or if a bug can be created.
• Bugs are fixed during debt week

[amunger]

If during a PR a test is failing, we have an 'owner' for each test. Owner decides if it needs to be fixed by PR or if a bug can be created.

@rchiodo - Would this require actually assigning out ownership to everyone on the team? From the discussion, I thought your idea was to just have the PR author investigate

[rchiodo]

I think the PR author should always investigate first, but each test would have an 'owner' (hopefully in the comments for the test) that could be pinged to ask what the owner thought should be done.

[rebornix]

I think the PR author should always investigate first, but each test would have an 'owner' (hopefully in the comments for the test)

This can also start with git blame if we don't know the author/owner for a specific test.

[DonJayamanne]

can also start with git blame if we don't

i agree, adding comments feels unnecessary. it should be easy to use git blame to find the author of a line of code (or test..) or go back in history and identify changes made by others