disk_snapshot_backup: init pod may get stuck due to concurrency call to Send

[YuJuncen]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Run a disk snapshot backup in a cluster that huge enough.
(Or just injecting some failpoints that make sometimes renewing the lease and sending wait apply happens concurrently.

2. What did you expect to see? (Required)

It should success to prepare -- nothing wrong happens.

3. What did you see instead (Required)

We were stuck at sending wait apply. When applying for quotas.

4. What is your TiDB version? (Required)

v6.5.x
But this may still happen in master.

NOTE

We call Send concurrently over a stream. Which isn't safe according to the document of ClientStream:

// It is safe to have a goroutine calling SendMsg and another goroutine
// calling RecvMsg on the same stream at the same time, but it is not safe
// to call SendMsg on the same stream in different goroutines. It is also
// not safe to call CloseSend concurrently with SendMsg.
//
// It is not safe to modify the message after calling SendMsg. Tracing
// libraries and stats handlers may use the message lazily.
SendMsg(m any) error