Flaky test list

[apostasie]

CI has been improved with the last few PRs, but there are still sources of flakyness.

Floodgates are open here: #3189

This ticket is to list them all, and get to the bottom of it and fix them all.

Feel free to add here any transient failure you see in the comments, with a copy of the logs and a link to the failing build (first failure in the run ONLY).

Fixed by #3192:

• TestCreateWithMACAddress - fails regularly - no idea - container_create_linux_test.go:121: assertion failed: res.ExitCode is not 0: Command should have succeeded%!(EXTRA string=time="2024-07-05T16:36:12Z" level=fatal msg="container \"e06fb05e3174c1be8afedfc8d58981cdf90f14507b5d7040e6e6f2448823b7ca\" in namespace \"nerdctl-test\": not found" ) see https://github.com/containerd/nerdctl/actions/runs/9811180349/job/27092899863?pr=3178
• TestExecEnv on windows - container_exec_test.go:79: assertion failed: res.ExitCode is not exitCode: time="2024-07-05T16:54:47Z" level=fatal msg="container \"30bbda4628649e094fee7f172c3aa7c16b1ddb032685f46bc297c9126fa9cefd\" in namespace \"nerdctl-test\": not found" - looks strangely similar - not sure what is going on here - is there some concurrent prune happening? https://github.com/containerd/nerdctl/actions/runs/9811363482/job/27093391801?pr=3181
• TestRunUserGID 2024-07-05T17:26:39Z" level=fatal msg="container \"d5eddc278faf40a910638e34c9186d5d1b4027c77be4e08f38192850bb194190\" in namespace \"nerdctl-test\": not found (https://github.com/containerd/nerdctl/actions/runs/9811583654/job/27093967186?pr=3178)

[apostasie]

These issues look a lot like #3092

Something really weird is going on.

[apostasie]

Canary PR: 10 failures

• TestTailFollowRotateLogs: 2 counts - this is very likely fixed by changes in Move to 24.04 for docker tests #3178
• TestIPFSAddress: 1 count - probably fixed by Fix IPFS container start routine #3181
• TestNetworkWithNamespace: 4 counts - task not found 3 times - possibly another variant of Successive calls to client.Containers(ctx) then c.Labels may fail with ErrNotFound (possibly other methods as well) #3167 - last count gives container not created: not found which is a bit more worrisome as this is from containerd
  • task not found likely fixed by Network prune should not fail if containers vanish #3194
• TestLogin: 1 count - TLS handshake timeout - either some weird connectivity issue, a crash in the auth container, or ?
• TestImageEncryptJWE: 1 count - unclear - image_encrypt_linux_test.go:127: assertion failed: res.ExitCode is 0
• TestComposeUp: 1 count - unclear - OCI runtime start failed: cannot start a container that has stopped: unknown

Build link:

https://github.com/containerd/nerdctl/actions/runs/9815148442?pr=3189

[apostasie]

Canary PR: 7 failures (3 counts TestTailFollowRotateLogs, 1 count TestIPFSAddress).

3 "new":

• TestBuildAttestation 2 counts - log is very weak - error: failed to solve: not found sounds like a buildkit message to me
• TestRunRestart - OCI runtime start failed: cannot start a container that has stopped: unknown - a bit worrisome - probably the same underlying issue as TestComposeUp above

Build link:

https://github.com/containerd/nerdctl/actions/runs/9815520719?pr=3189

[apostasie]

Canary: 7 failures

• TestRunWithSystemdTrueEnabled
• TestBuildAttestation
• TestImageEncryptJWE
• TestTailFollowRotateLogs
• TestIPFSComposeUp (2 counts)
• TestVolumeLsFilter

[apostasie]

TestBuildAttestation & TestTailFollowRotateLogs are now top two.

[apostasie]

As of today, a lot of the issues got PR merged-in.

Latest Canary build (with extra fixes for TestBuildAttestation) failed 4 times:

• TestCopyToContainer
• TestTailFollowRotateLogs is still failing (3x count)

[apostasie]

Latest (unmerged yet) version of the test tooling will introduce a way to mark flaky tests in code - allowing to have separate runs for flaky (with retry) / not flaky.

This ticket here is getting outdated and is unwildly.

Closing for the better solution of in-code marking.