TiDB sends too much pings to PD follower occasionally

[BusyJay]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Use tiup to setup a cluster with 3 PD + 3 TiKV + 4 TiDB.
2. Wait some time.
3. You can see logs like following in one of the PD followers:

[2020/06/09 16:17:30.527 +08:00] [ERROR] [grpclog.go:75] ["transport: Got too many pings from the client, closing the connection."]
[2020/06/09 16:17:30.527 +08:00] [ERROR] [grpclog.go:75] ["transport: loopyWriter.run returning. Err: transport: Connection closing"]
[2020/06/09 16:17:30.528 +08:00] [WARN] [grpclog.go:60] ["transport: http2Server.HandleStreams failed to read frame: read tcp 172.16.4.218:2379->172.16.4.130:49252: use of closed network connection"]
[2020/06/09 16:17:30.534 +08:00] [ERROR] [grpclog.go:75] ["transport: Got too many pings from the client, closing the connection."]
[2020/06/09 16:17:30.534 +08:00] [ERROR] [grpclog.go:75] ["transport: loopyWriter.run returning. Err: transport: Connection closing"]
[2020/06/09 16:17:30.534 +08:00] [WARN] [grpclog.go:60] ["transport: http2Server.HandleStreams failed to read frame: read tcp 172.16.4.218:2379->172.16.4.163:58510: use of closed network connection"]
[2020/06/09 16:17:30.550 +08:00] [ERROR] [grpclog.go:75] ["transport: Got too many pings from the client, closing the connection."]
[2020/06/09 16:17:30.550 +08:00] [ERROR] [grpclog.go:75] ["transport: loopyWriter.run returning. Err: transport: Connection closing"]
[2020/06/09 16:17:30.550 +08:00] [WARN] [grpclog.go:60] ["transport: http2Server.HandleStreams failed to read frame: read tcp 172.16.4.218:2379->172.16.4.119:37806: use of closed network connection"]
[2020/06/09 16:17:30.568 +08:00] [ERROR] [grpclog.go:75] ["transport: Got too many pings from the client, closing the connection."]
[2020/06/09 16:17:30.568 +08:00] [ERROR] [grpclog.go:75] ["transport: loopyWriter.run returning. Err: transport: Connection closing"]
[2020/06/09 16:17:30.568 +08:00] [WARN] [grpclog.go:60] ["transport: http2Server.HandleStreams failed to read frame: read tcp 172.16.4.218:2379->172.16.4.218:34962: use of closed network connection"]
[2020/06/09 16:38:50.523 +08:00] [ERROR] [grpclog.go:75] ["transport: Got too many pings from the client, closing the connection."]
[2020/06/09 16:38:50.523 +08:00] [ERROR] [grpclog.go:75] ["transport: loopyWriter.run returning. Err: transport: Connection closing"]
[2020/06/09 16:38:50.523 +08:00] [WARN] [grpclog.go:60] ["transport: http2Server.HandleStreams failed to read frame: read tcp 172.16.4.218:2379->172.16.4.130:50278: use of closed network connection"]
[2020/06/09 16:38:50.536 +08:00] [ERROR] [grpclog.go:75] ["transport: Got too many pings from the client, closing the connection."]
[2020/06/09 16:38:50.536 +08:00] [ERROR] [grpclog.go:75] ["transport: loopyWriter.run returning. Err: transport: Connection closing"]
[2020/06/09 16:38:50.536 +08:00] [WARN] [grpclog.go:60] ["transport: http2Server.HandleStreams failed to read frame: read tcp 172.16.4.218:2379->172.16.4.163:59352: use of closed network connection"]
[2020/06/09 16:38:50.552 +08:00] [ERROR] [grpclog.go:75] ["transport: Got too many pings from the client, closing the connection."]
[2020/06/09 16:38:50.552 +08:00] [ERROR] [grpclog.go:75] ["transport: loopyWriter.run returning. Err: transport: Connection closing"]
[2020/06/09 16:38:50.552 +08:00] [WARN] [grpclog.go:60] ["transport: http2Server.HandleStreams failed to read frame: read tcp 172.16.4.218:2379->172.16.4.119:39066: use of closed network connection"]
[2020/06/09 16:38:50.570 +08:00] [ERROR] [grpclog.go:75] ["transport: Got too many pings from the client, closing the connection."]
[2020/06/09 16:38:50.570 +08:00] [ERROR] [grpclog.go:75] ["transport: loopyWriter.run returning. Err: transport: Connection closing"]
[2020/06/09 16:38:50.570 +08:00] [WARN] [grpclog.go:60] ["transport: http2Server.HandleStreams failed to read frame: read tcp 172.16.4.218:2379->172.16.4.218:35778: use of closed network connection"]

2. What did you expect to see? (Required)

No error is reported.

3. What did you see instead (Required)

Disconnections occasionally.

4. Affected version (Required)

v4.0.0

5. Root Cause Analysis

[HunDunDM]

There are multiple PermitWithoutStream that need to be modified. And in TiKV server-side, PermitWithoutStream is false too.