PD enters deadloop in key manager watch routine and costs 100% CPU

[amyangfei]

Bug Report

What did you do?

Run a simple PD cluster with 3 nodes.

What did you expect to see?

PD runs normally

What did you see instead?

One of the PDs is abnormal with CPU usage (172.16.6.135:12379 in the following monitor), from the perf we can observe server.(*Server).encryptionKeyManagerLoop costs most of the CPU

The root cause is after the etcd watch is canceled(etcd server returns an error, such as etcdserver: mvcc: required revision has been compacted), the watched channel is closed and will always get nil value from it.

This log can be found in abnormal pd.

[2021/08/28 13:02:21.352 +08:00] [WARN] [key_manager.go:226] ["encryption key watcher canceled"]

dead loop here:

pd/server/encryptionkm/key_manager.go

Lines 224 to 241 in fa09aff

	case resp := <-watchChan:
	if resp.Canceled {
	// If the watcher failed, we fallback to reload every 10 minutes.
	log.Warn("encryption key watcher canceled")
	watcherEnabled = false
	continue
	}
	for _, event := range resp.Events {
	if event.Type != mvccpb.PUT {
	log.Warn("encryption keys is deleted unexpectedly")
	continue
	}
	_, err := m.loadKeysFromKV(event.Kv)
	if err != nil {
	log.Warn("fail to get encryption keys from watcher result", errs.ZapError(err))
	}
	}
	m.helper.eventAfterReloadByWatcher()

What version of PD are you using (pd-server -V)?

Release Version: v5.1.0
Edition: Community
Git Commit Hash: 8bc9675a923f81f79d8a566e208c8afdcf4ea3f3
Git Branch: heads/refs/tags/v5.1.0
UTC Build Time:  2021-06-24 07:09:49

master, v5.2.0 and v5.0.0 are also affected