deadlock (?) in DM UT

[lance6716]

Which jobs are flaking?

DM UT

Which test(s) are flaking?

dm/master

Jenkins logs or GitHub Actions link

https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/16265/pipeline/

Anything else we need to know

• Does this test exist for other branches as well?

• Has there been a high frequency of failure lately?

[lance6716]

dm/dm/master (5m0.554s) in CI

and my PC dm/dm/master also fail once

[D3Hunter]

met again here:
https://ci.pingcap.net/blue/rest/organizations/jenkins/pipelines/ut-check/runs/16305/nodes/45/steps/49/log/?start=0

[lance6716]

still 🤔

[2022-06-13T02:14:23.985Z] goroutine 18941 [semacquire]:

[2022-06-13T02:14:23.985Z] sync.runtime_Semacquire(0xc0010bc278?)

[2022-06-13T02:14:23.985Z] 	/usr/local/go/src/runtime/sema.go:56 +0x25

[2022-06-13T02:14:23.985Z] sync.(*WaitGroup).Wait(0xc0010bc270)

[2022-06-13T02:14:23.985Z] 	/usr/local/go/src/sync/waitgroup.go:136 +0x9c

[2022-06-13T02:14:23.985Z] github.com/pingcap/tiflow/dm/pkg/election.(*Election).Close(0xc0010bc210)

[2022-06-13T02:14:23.985Z] 	/home/jenkins/agent/workspace/ut-check/tiflow/dm/pkg/election/election.go:197 +0x1e7

[2022-06-13T02:14:23.985Z] github.com/pingcap/tiflow/dm/dm/master.(*Server).Close(0xc003acc000)

[2022-06-13T02:14:23.986Z] 	/home/jenkins/agent/workspace/ut-check/tiflow/dm/dm/master/server.go:293 +0x1a5

[2022-06-13T02:14:23.986Z] github.com/pingcap/tiflow/dm/dm/master.(*testMaster).TestOfflineMember.func1()

[2022-06-13T02:14:23.986Z] 	/home/jenkins/agent/workspace/ut-check/tiflow/dm/dm/master/server_test.go:1915 +0x45

[2022-06-13T02:14:23.986Z] github.com/pingcap/tiflow/dm/dm/master.(*testMaster).TestOfflineMember(0xc001445bf0, 0xc001eb50e0)

[2022-06-13T02:14:23.986Z] 	/home/jenkins/agent/workspace/ut-check/tiflow/dm/dm/master/server_test.go:2062 +0x245e

https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/16799/pipeline/

[lance6716]

maybe etcd close session is too slow because etcd members are stopped?

[2022-06-13T02:14:23.992Z] goroutine 19518 [select]:

[2022-06-13T02:14:23.992Z] google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc00349f320)

[2022-06-13T02:14:23.992Z] 	/go/pkg/mod/google.golang.org/grpc@v1.46.2/internal/transport/transport.go:324 +0x10e

[2022-06-13T02:14:23.992Z] google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)

[2022-06-13T02:14:23.992Z] 	/go/pkg/mod/google.golang.org/grpc@v1.46.2/internal/transport/transport.go:339

[2022-06-13T02:14:23.992Z] google.golang.org/grpc.(*csAttempt).recvMsg(0xc003498840, {0x61bf940?, 0xc002f776e0}, 0x0)

[2022-06-13T02:14:23.992Z] 	/go/pkg/mod/google.golang.org/grpc@v1.46.2/stream.go:980 +0x13e

[2022-06-13T02:14:23.992Z] google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc00349f18c?)

[2022-06-13T02:14:23.992Z] 	/go/pkg/mod/google.golang.org/grpc@v1.46.2/stream.go:845 +0x57

[2022-06-13T02:14:23.992Z] google.golang.org/grpc.(*clientStream).withRetry(0xc00349f0e0, 0xc0021cae70, 0xc0021cae40)

[2022-06-13T02:14:23.992Z] 	/go/pkg/mod/google.golang.org/grpc@v1.46.2/stream.go:709 +0xd7

[2022-06-13T02:14:23.992Z] google.golang.org/grpc.(*clientStream).RecvMsg(0xc00349f0e0, {0x61bf940, 0xc002f776e0})

[2022-06-13T02:14:23.992Z] 	/go/pkg/mod/google.golang.org/grpc@v1.46.2/stream.go:844 +0x186

[2022-06-13T02:14:23.992Z] google.golang.org/grpc.invoke({0x78ee970, 0xc002f77740}, {0x63c766b, 0x1f}, {0x61bf800, 0xc002f776b0}, {0x61bf940, 0xc002f776e0}, 0x5f?, {0xc002e42f80, ...})

[2022-06-13T02:14:23.992Z] 	/go/pkg/mod/google.golang.org/grpc@v1.46.2/call.go:73 +0x14e

[2022-06-13T02:14:23.992Z] go.etcd.io/etcd/client/v3.(*Client).unaryClientInterceptor.func1({0x78ee938, 0xc001d21380}, {0x63c766b, 0x1f}, {0x61bf800, 0xc002f776b0}, {0x61bf940, 0xc002f776e0}, 0xc00071c700, 0x64dc988, ...)

[2022-06-13T02:14:23.992Z] 	/go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.2/retry_interceptor.go:58 +0x51c

[2022-06-13T02:14:23.992Z] google.golang.org/grpc.(*ClientConn).Invoke(0xc00071c700, {0x78ee938, 0xc001d21380}, {0x63c766b, 0x1f}, {0x61bf800, 0xc002f776b0}, {0x61bf940, 0xc002f776e0}, {0xc001d213e0, ...})

[2022-06-13T02:14:23.992Z] 	/go/pkg/mod/google.golang.org/grpc@v1.46.2/call.go:35 +0x26c

[2022-06-13T02:14:23.993Z] go.etcd.io/etcd/api/v3/etcdserverpb.(*leaseClient).LeaseRevoke(0xc0007aaa88, {0x78ee938, 0xc001d21380}, 0x30?, {0xc001d213e0, 0x4, 0x6})

[2022-06-13T02:14:23.993Z] 	/go/pkg/mod/go.etcd.io/etcd/api/v3@v3.5.2/etcdserverpb/rpc.pb.go:6821 +0xf5

[2022-06-13T02:14:23.993Z] go.etcd.io/etcd/client/v3.(*retryLeaseClient).LeaseRevoke(0xc00122d370, {0x78ee938, 0xc001d21380}, 0x170b0e5?, {0xa00fb20?, 0x3, 0x3})

[2022-06-13T02:14:23.993Z] 	/go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.2/retry.go:148 +0x1ca

[2022-06-13T02:14:23.993Z] go.etcd.io/etcd/client/v3.(*lessor).Revoke(0xc0033ec6e0, {0x78ee938, 0xc001d21380}, 0xaec815ad34f5604)

[2022-06-13T02:14:23.993Z] 	/go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.2/lease.go:231 +0x137

[2022-06-13T02:14:23.993Z] go.etcd.io/etcd/client/v3/concurrency.(*Session).Close(0xc0011e75c0)

[2022-06-13T02:14:23.993Z] 	/go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.2/concurrency/session.go:99 +0x155

[2022-06-13T02:14:23.993Z] github.com/pingcap/tiflow/dm/pkg/election.(*Election).campaignLoop.func1(0xc0011e75c0)

[2022-06-13T02:14:23.993Z] 	/home/jenkins/agent/workspace/ut-check/tiflow/dm/pkg/election/election.go:203 +0x54

[2022-06-13T02:14:23.993Z] github.com/pingcap/tiflow/dm/pkg/election.(*Election).campaignLoop.func3()

[2022-06-13T02:14:23.993Z] 	/home/jenkins/agent/workspace/ut-check/tiflow/dm/pkg/election/election.go:217 +0x87

[lance6716]

https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/17216/pipeline

meet again