Improve Memory Management

[zz-jason]

Description

Since TiDB 2.1, we have supported memory tracking to track the memory usage of a query. In the recent releases, we have supported canceling a query which consumes too much memory or spilling the intermediate result to disk for a query to reduce the memory usage.

But there's still lots of improvements related to memory management, for example:

• Reduce memory usage for a query without spilling disk. A goal of this can be set to pass all the TPC-H (SF=10) queries without spilling disk in a 16 GB environment.
• Optimize memory usages for internal components, like statistics cache(reduce memory consumption of stats data #16572), table reader, coprocessor client.
• Make more operators be able to spill intermediate results to disk: Table Readers, Hash Aggregate, Insert/Update/Delete/Replace, etc.
• Implement instance-level memory control, which needs careful design.
• 10GB transaction memory usage optimization

See issues labeled with epic/memory-management for details.

P0 Task List

• Reduce the memory usage of huge transaction #18308 Enlarge Transaction Size Limitation And Prevent OOM
• reduce memory consumption of stats data #16572 Reduce Memory Consumption of Stats Data
• mydumper OOM when dumping a 200 GB database from TiDB mydumper#39 Solve the mydumper OOM Issue, which is caused by TableReader consumes too much memory #16104
• Encrypt the Data Spill to Disk

Category

Performance

Value

• Value Score: 5

Lots of users have reported memory issues, such as can not cancel the query consumes too much memory, which is caused by the memory of some components are tracked. (priority/P1)

In the DBaaS environment, the memory usually is not that rich, reducing the memory usage without spilling disk or enable more operators to spill disk can greatly improve the usability. (priority/P1)

Usually, there's a lot of tables on a cluster(more than 1 million tables), reducing the statistics cache can heavily reduce the overall memory consumption of the tidb process, which further reduces the OOM risk. (priority/P0)

Workload Estimation

Workload contains coding, documenting, and testing.
1 Point for 1 Person/Work Day

• 150 Points

Time

GanttStart: 2020-07-01
GanttDue: 2020-09-30
GanttProgress: 50%