tidb-server refuses to start if no pump is registered with pd

[kolbe]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do?
   When trying to restart all TiDB cluster components simultaneously, tidb-server will exit if binlog is enabled but no pump instances are yet registered with pd.

2. What did you expect to see?
   tidb-server should continue looking for a pump instance either indefinitely or until some timeout, depending on prevailing wisdom and consensus about startup behavior of other components.

3. What did you see instead?

2019/04/23 13:10:36.520 terror.go:329: [fatal] no pump found in pd
github.com/pingcap/tidb-tools/tidb-binlog/pump_client.NewPumpsClient
        /home/jenkins/workspace/build_tidb_master/go/pkg/mod/github.com/pingcap/tidb-tools@v2.1.3-0.20190321065848-1e8b48f5c168+incompatible/tidb-binlog/pump_client/client.go:168
main.setupBinlogClient
        /home/jenkins/workspace/build_tidb_master/go/src/github.com/pingcap/tidb/tidb-server/main.go:227
main.main
        /home/jenkins/workspace/build_tidb_master/go/src/github.com/pingcap/tidb/tidb-server/main.go:166
runtime.main
        /usr/local/go/src/runtime/proc.go:200
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1337
[4]+  Exit 1                  ./bin/tidb-server --config=tidb.toml &>tidb.out

4. What version of TiDB are you using (tidb-server -V or run select tidb_version(); on TiDB)?

Release Version: v3.0.0-beta.1-150-g9c8396e78
Git Commit Hash: 9c8396e789dfa8681ae290a3f1f96cdedffc320e
Git Branch: master
UTC Build Time: 2019-04-23 03:11:16
GoVersion: go version go1.12 linux/amd64
Race Enabled: false
TiKV Min Version: 2.1.0-alpha.1-ff3dd160846b7d1aed9079c389fc188f7f5ea13e
Check Table Before Drop: false

[zz-jason]

@amyangfei PTAL

[lonng]

Hi, @kolbe This error occurred because of your deployment and configuration inconsistent. Pump instances must be available if enable-binlog is true.

Maybe we should refine the error message and make it more clear.

[amyangfei]

yes, if we configure TiDB with enable-binlog = true, the recommended startup sequence is: PD -> TiKV -> Pump -> TiDB -> Drainer

[morgo]

yes, if we configure TiDB with enable-binlog = true, the recommended startup sequence is: PD -> TiKV -> Pump -> TiDB -> Drainer

TiKV and TiDB have been improved so that they will retry connecting to pd if they are first to start. This helps simplify deployment since guaranteeing order adds complexity to systems. For example see: tikv/tikv#4500

The behavior change that @kolbe is suggesting makes sense to me. The TiDB server should block in a retry loop waiting for a pump server to be available instead of hard failing.