master: pd panic during running workload with multiple resource groups

[mayjiang0203]

Bug Report

What did you do?

1. create 1000 resource groups and assign them to 1000 different users.
2. run workload with 32 different users.

What did you expect to see?

No errors report.

What did you see instead?

One pd panic.

{"namespace":"e2e-oltp-multiple-resource-group-tps-2071148-1-37","level":"FATAL","container":"pd","log":"[log.go:87] [panic] [recover={}] [stack=\"[github.com/tikv/pd/pkg/utils/logutil.LogPanic\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/utils/logutil/log.go:87\\nruntime.gopanic\\n\\t/usr/local/go/src/runtime/panic.go:914\\ngithub.com/prometheus/client_golang/prometheus.(*counter).Add\\n\\t/go/pkg/mod/github.com/prometheus/client_golang@v1.11.1/prometheus/counter.go:109\\ngithub.com/tikv/pd/pkg/mcs/resourcemanager/server.(*Manager).backgroundMetricsFlush\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/mcs/resourcemanager/server/manager.go:327](http://github.com/tikv/pd/pkg/utils/logutil.LogPanic//n//t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/utils/logutil/log.go:87//nruntime.gopanic//n//t/usr/local/go/src/runtime/panic.go:914//ngithub.com/prometheus/client_golang/prometheus.(*counter).Add//n//t/go/pkg/mod/github.com/prometheus/client_golang@v1.11.1/prometheus/counter.go:109//ngithub.com/tikv/pd/pkg/mcs/resourcemanager/server.(*Manager).backgroundMetricsFlush//n//t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/mcs/resourcemanager/server/manager.go:327)\"]","pod":"tc-pd-0"}

What version of PD are you using (pd-server -V)?

./pd-server -V
 Release Version: v7.4.0-alpha
Edition: Community
Git Commit Hash: 346e7716e2598dfba2db6afa73c3731e15449f49
Git Branch: heads/refs/tags/v7.4.0-alpha
UTC Build Time:  2023-08-16 11:36:22

[mayjiang0203]

/assign @CabinfeverB
/severity Critical