Description
Description
Since TiDB 2.1, we have supported memory tracking to track the memory usage of a query. In the recent releases, we have supported canceling a query which consumes too much memory or spilling the intermediate result to disk for a query to reduce the memory usage.
But there's still lots of improvements related to memory management, for example:
- Reduce memory usage for a query without spilling disk. A goal of this can be set to pass all the TPC-H (SF=10) queries without spilling disk in a 16 GB environment.
- Optimize memory usages for internal components, like statistics cache(reduce memory consumption of stats data #16572), table reader, coprocessor client.
- Make more operators be able to spill intermediate results to disk: Table Readers, Hash Aggregate, Insert/Update/Delete/Replace, etc.
- Implement instance-level memory control, which needs careful design.
- 10GB transaction memory usage optimization
See issues labeled with epic/memory-management for details.
P0 Task List
- Reduce the memory usage of huge transaction #18308 Enlarge Transaction Size Limitation And Prevent OOM
- reduce memory consumption of stats data #16572 Reduce Memory Consumption of Stats Data
- mydumper OOM when dumping a 200 GB database from TiDB mydumper#39 Solve the mydumper OOM Issue, which is caused by TableReader consumes too much memory #16104
- Encrypt the Data Spill to Disk
Category
Performance
Value
- Value Score: 5
Lots of users have reported memory issues, such as can not cancel the query consumes too much memory, which is caused by the memory of some components are tracked. (priority/P1)
In the DBaaS environment, the memory usually is not that rich, reducing the memory usage without spilling disk or enable more operators to spill disk can greatly improve the usability. (priority/P1)
Usually, there's a lot of tables on a cluster(more than 1 million tables), reducing the statistics cache can heavily reduce the overall memory consumption of the tidb process, which further reduces the OOM risk. (priority/P0)
Workload Estimation
Workload contains coding, documenting, and testing.
1 Point for 1 Person/Work Day
- 150 Points
Time
GanttStart: 2020-07-01
GanttDue: 2020-09-30
GanttProgress: 50%