Incubating Project: TiDB over RoCE

[raygift]

Project Incubating Request

TiDB-over-RoCE

Describe the project you want to incubate:

Currently, message transmissions inside TiDB such as raft heartbeat RPCs are based on TCP. Specifically, TiDB server and PD using grpc-go , while TiKV using gRPC Core wrapped by gRPC-rs, both of these gRPC projects using TCP as the underlying communication protocol.

RDMA(Remote Direct Memory Access) allows one machine to read or write to the memory of another machine without requiring any processor cycles on the target. It has been paid more attention since its especial advantages such as Kernel-Bypass and Zero-Copy. As a RDMA protocol based on Ethernet, RoCEv2 (RDMA over Converged Ethernet v2) achieves a very well balance between efficiency of network communication and compatibility in existing data center infrastructure.

This proposal proposes running TiDB over RoCE protocol, which means all communications between tidb server, pd and tikv can benefit from RDMA technology, and higher performance and lower CPU workload are expected.

Estimated Time

10 months

Initial Team Members

张培承(raygift)
李毅伦(typedefstruct32)

Your RFC/Proposal?

[winkyao]

LGTM

[niubell]

LGTM

[sunxiaoguang]

LGTM

[sunxiaoguang]

The project is very likely to be tightly coupled with gRPC implementation for both Rust and Golang. Therefore you may have to spend lots of efforts to make significant improvements to convenience people to accept such changes. Another thing I personally would like to see in this project is RDMA in virtualized environment. Good luck:)

[winkyao]

@raygift https://github.com/tidb-incubator/TiDBOverRoCE have been created, already invited you to be the admin of this project. Please accept the invitation https://github.com/tidb-incubator/TiDBOverRoCE/invitations