Lots of "stream to store closed" log and CDC sends huge pending tasks to TiKV which results in TiKV OOM

[fubinzh]

What did you do?

This issue is created from tikv/tikv#16035, to track the CDC side issue that lots of "stream to store closed"
and CDC send huge pending tasks to TiKV which results TiKV OOM.
While TiKV side fix will limit the pending initial scan tasks, cdc side need further investigate the "stream to store closed" issue.

[2023/11/19 16:11:13.208 +08:00] [INFO] [client.go:981] ["stream to store closed"] [namespace=default] [changefeed=test1] [addr=tc-tikv-2.tc-tikv-peer.webank-cdc-tps-4590005-1-42.svc:20160] [storeID=8]

What did you expect to see?

CDC should not send so many tasks to TiKV

What did you see instead?

CDC send huge pending tasks to TiKV which results TiKV OOM.

Versions of the cluster

Upstream TiKV version (execute tikv-server --version):

/ # /tikv-server -V                                                                                                                            
TiKV                                                                                                                                           
Release Version:   6.5.3                                                                                                                       
Edition:           Enterprise                                                                                                                  
Git Commit Hash:   5165943fbffce298b61698a59620f415df239420                                                                                    
Git Commit Branch: heads/refs/tags/v6.5.3-20231116-5165943                                                                                     
UTC Build Time:    2023-11-16 07:07:20                                                                                                         
Rust Version:      rustc 1.67.0-nightly (96ddd32c4 2022-11-14)                                                                                 
Enable Features:   pprof-fp jemalloc mem-profiling portable test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure
Profile:           dist_release                                                                                                                

TiCDC version (execute cdc version):

bash-5.1# /cdc version                                    
Release Version: v6.5.3-20231116-255d810                  
Git Commit Hash: 255d8104c8b9b4f8af191ee69d301ced723ccc3d 
Git Branch: heads/refs/tags/v6.5.3-20231116-255d810       
UTC Build Time: 2023-11-16 06:35:28                       
Go Version: go version go1.19.12 linux/amd64              
Failpoint Build: false                                    

[fubinzh]

/severity moderate

[hicqu]

Closed by #10237