gRPC initial window is too large that may cause OOM

[overvenus]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.

Capture 10k tables with ~90G incremental data using single CDC node.

heap profile: profile.pb.gz

gRPC client	flamegraph

2. What did you expect to see?

No OOM.

3. What did you see instead?

gRPC client consumes too many memory and cause OOM.

4. Versions of the cluster

v5.1.0

[overvenus]

This issue relates to #2553

[amyangfei]

Data accumulated in kv client, it may be caused by low throughput of gRPC message processing. Is there any CPU profile dump during the test.

[overvenus]

Data accumulated in kv client, it may be caused by low throughput of gRPC message processing.

Yes, data accumulation is caused by unbalanced producing and consuming speed, in this case, we have a slow sorter (i/o bottleneck), kv client is much faster than the sorter.

Also, after changing initial window size to 64KB and initial conn window size to 8MB, the OOM disappeared.

-	grpcInitialWindowSize     = 1 << 26 // 64 MB The value for initial window size on a stream
-	grpcInitialConnWindowSize = 1 << 27 // 128 MB The value for initial window size on a connection
+	grpcInitialWindowSize     = 65535 // 64 KB The value for initial window size on a stream
+	grpcInitialConnWindowSize = 1 << 23 // 8 MB The value for initial window size on a connection

[overvenus]

Is there any CPU profile dump during the test.

profile.pb.gz

[lonng]

Data accumulated in kv client, it may be caused by low throughput of gRPC message processing.

Yes, data accumulation is caused by unbalanced producing and consuming speed, in this case, we have a slow sorter (i/o bottleneck), kv client is much faster than the sorter.

Also, after changing initial window size to 64KB and initial conn window size to 8MB, the OOM disappeared.

-	grpcInitialWindowSize     = 1 << 26 // 64 MB The value for initial window size on a stream
-	grpcInitialConnWindowSize = 1 << 27 // 128 MB The value for initial window size on a connection
+	grpcInitialWindowSize     = 65535 // 64 KB The value for initial window size on a stream
+	grpcInitialConnWindowSize = 1 << 23 // 8 MB The value for initial window size on a connection

Is there performance issue if we lower those values?