[BUG] All CI jobs timing out

[invidian]

It seems runner image got updated recently and things start to fail. I see this now in docker logs in runner:

Feb 18 14:50:25 runnervmn36qa dockerd[4236]: fatal error: concurrent map writes
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: goroutine 393790 [running]:
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: internal/runtime/maps.fatal({0x5649a38a4931?, 0xc00214cc80?})
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /usr/local/go/src/runtime/panic.go:1046 +0x18
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/buildkit/sourcepolicy.(*wildcardCache).Match(0xc0073333f0, {0xc00214cc80, 0xf})
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/moby/buildkit/sourcepolicy/formatter.go:66 +0xbf
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/buildkit/sourcepolicy.match(0xc002814630, {0xc00214cc80, 0xf}, {0x0, 0x0, 0xc001fe9e30?}, 0xc00657c2d0)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/moby/buildkit/sourcepolicy/matcher.go:56 +0x48f
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/buildkit/sourcepolicy.(*Engine).evaluatePolicy(0xc00657c210, {0x5649a45e89b8, 0xc004333f20}, 0xc0034ec730, 0xc003cf00c0)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/moby/buildkit/sourcepolicy/engine.go:133 +0x305
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/buildkit/sourcepolicy.(*Engine).evaluatePolicies(0xc00657c210, {0x5649a45e89b8, 0xc004333f20}, 0xc003cf00c0)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/moby/buildkit/sourcepolicy/engine.go:101 +0x7a
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/buildkit/sourcepolicy.(*Engine).Evaluate(0xc00657c210, {0x5649a45e89f0, 0xc0034ec6e0}, 0xc003cf00c0)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/moby/buildkit/sourcepolicy/engine.go:84 +0x312
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/buildkit/solver/llbsolver.(*policyEvaluator).Evaluate(0xc007333180, {0x5649a45e89f0, 0xc0034ec6e0}, 0xc003ef9080)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/moby/buildkit/solver/llbsolver/policy.go:28 +0x90
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/buildkit/solver/llbsolver.loadLLB.func1()
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/moby/buildkit/solver/llbsolver/vertex.go:353 +0x6e
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: golang.org/x/sync/errgroup.(*Group).Go.func1()
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/golang.org/x/sync/errgroup/errgroup.go:93 +0x50
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 393788
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x95
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: goroutine 1 [sync.WaitGroup.Wait, 14 minutes, locked to thread]:
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: sync.runtime_SemacquireWaitGroup(0xc000b05e60?, 0x80?)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /usr/local/go/src/runtime/sema.go:114 +0x2e
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: sync.(*WaitGroup).Wait(0xc000b669e0)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /usr/local/go/src/sync/waitgroup.go:206 +0x85
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/moby/v2/daemon/command.(*daemonCLI).start(0xc000518af0, {0x5649a45e8980, 0x5649a5e42380})
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/daemon/command/daemon.go:349 +0x1b25
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/moby/v2/daemon/command.runDaemon(...)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/daemon/command/docker_unix.go:13
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/moby/v2/daemon/command.newDaemonCommand.func1(0xc00048c908, {0xc0006e4210?, 0x7?, 0x5649a385c4a3?})
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/daemon/command/docker.go:48 +0xb3
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/spf13/cobra.(*Command).execute(0xc00048c908, {0xc0000520d0, 0x3, 0x3})
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/spf13/cobra/command.go:1015 +0xb02
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/spf13/cobra.(*Command).ExecuteC(0xc00048c908)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/spf13/cobra/command.go:1148 +0x465
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/spf13/cobra.(*Command).Execute(...)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/spf13/cobra/command.go:1071
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/spf13/cobra.(*Command).ExecuteContext(...)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/spf13/cobra/command.go:1064
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/moby/v2/daemon/command.daemonRunner.Run({0x5649a45b4df8?}, {0x5649a45e8980, 0x5649a5e42380})
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/daemon/command/docker.go:97 +0x6e
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: main.main()
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/cmd/dockerd/main.go:35 +0x119

[invidian]

@cpuguy83 pointed that this is caused by moby/buildkit@f7682fe, so disabling source policy on CI should fix it and it seems to be working. I'll create a PR.