Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kv: reduce gRPC window size to avoid OOM (#2699) #2724

Merged
merged 6 commits into from
Sep 3, 2021
Merged

kv: reduce gRPC window size to avoid OOM (#2699) #2724

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
0211711
kv: reduce gRPC window size to avoid OOM
overvenus Sep 1, 2021
ca6cbc5
go fmt
overvenus Sep 1, 2021
045b8f0
address comments
overvenus Sep 2, 2021
2a3dd81
Merge branch 'release-5.0' into cherry-pick-2699-to-release-5.0
ti-chi-bot Sep 2, 2021
f911b5b
Merge branch 'release-5.0' into cherry-pick-2699-to-release-5.0
ti-chi-bot Sep 2, 2021
03f43fe
Merge branch 'release-5.0' into cherry-pick-2699-to-release-5.0
ti-chi-bot Sep 2, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions cdc/kv/client.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,16 @@ import (
const (
dialTimeout = 10 * time.Second
tikvRequestMaxBackoff = 20000 // Maximum total sleep time(in ms)
// TODO find optimal values and test extensively before releasing
// The old values cause the gRPC stream to stall for some unknown reason.
grpcInitialWindowSize = 1 << 26 // 64 MB The value for initial window size on a stream
grpcInitialConnWindowSize = 1 << 27 // 128 MB The value for initial window size on a connection
grpcMaxCallRecvMsgSize = 1 << 28 // 256 MB The maximum message size the client can receive

// TiCDC may open numerous gRPC streams,
// with 65535 bytes window size, 10K streams takes about 27GB memory.
//
// 65535 bytes, the initial window size in http2 spec.
grpcInitialWindowSize = (1 << 16) - 1
// 8 MB The value for initial window size on a connection
grpcInitialConnWindowSize = 1 << 23
// 256 MB The maximum message size the client can receive
grpcMaxCallRecvMsgSize = 1 << 28

// The threshold of warning a message is too large. TiKV split events into 6MB per-message.
warnRecvMsgSizeThreshold = 12 * 1024 * 1024
Expand Down