Skip to content

TiFlash resource control and pipeline execution model: update docs link and refine wording #14917

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions releases/release-7.2.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v7.2/quick-start-with-

* TiFlash supports the pipeline execution model (experimental) [#6518](https://github.com/pingcap/tiflash/issues/6518) @[SeaRise](https://github.com/SeaRise)

Prior to v7.2.0, each task in the TiFlash engine must individually request thread resources during execution. TiFlash controls the number of tasks to limit thread resource usage and prevent overuse, but this issue could not be completely eliminated. To address this problem, starting from v7.2.0, TiFlash introduces a pipeline execution model. This model centrally manages all thread resources and schedules task execution uniformly, maximizing the utilization of thread resources while avoiding resource overuse. To enable or disable the pipeline execution model, modify the [`tidb_enable_tiflash_pipeline_model`](https://docs.pingcap.com/zh/tidb/v7.2/system-variables#tidb_enable_tiflash_pipeline_model-new-in-v720) system variable.
Prior to v7.2.0, each task in the TiFlash engine must individually request thread resources during execution. TiFlash controls the number of tasks to limit thread resource usage and prevent overuse, but this issue could not be completely eliminated. To address this problem, starting from v7.2.0, TiFlash introduces a pipeline execution model. This model centrally manages all thread resources and schedules task execution uniformly, maximizing the utilization of thread resources while avoiding resource overuse. To enable or disable the pipeline execution model, modify the [`tidb_enable_tiflash_pipeline_model`](https://docs.pingcap.com/tidb/v7.2/system-variables#tidb_enable_tiflash_pipeline_model-new-in-v720) system variable.

For more information, see [documentation](/tiflash/tiflash-pipeline-model.md).

Expand Down Expand Up @@ -180,7 +180,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v7.2/quick-start-with-
| [`tidb_analyze_skip_column_types`](/system-variables.md#tidb_analyze_skip_column_types-new-in-v720) | Newly added | Controls which types of columns are skipped for statistics collection when executing the `ANALYZE` command to collect statistics. The variable is only applicable for [`tidb_analyze_version = 2`](/system-variables.md#tidb_analyze_version-new-in-v510). When using the syntax of `ANALYZE TABLE t COLUMNS c1, ..., cn`, if the type of a specified column is included in `tidb_analyze_skip_column_types`, the statistics of this column will not be collected. |
| [`tidb_enable_check_constraint`](/system-variables.md#tidb_enable_check_constraint-new-in-v720) | Newly added | Controls whether to enable `CHECK` constraints. The default value is `OFF`, which means this feature is disabled. |
| [`tidb_enable_fast_table_check`](/system-variables.md#tidb_enable_fast_table_check-new-in-v720) | Newly added | Controls whether to use a checksum-based approach to quickly check the consistency of data and indexes in a table. The default value is `ON`, which means this feature is enabled. |
| [`tidb_enable_tiflash_pipeline_model`](https://docs.pingcap.com/zh/tidb/v7.2/system-variables#tidb_enable_tiflash_pipeline_model-new-in-v720) | Newly added | Controls whether to enable the new execution model of TiFlash, the [pipeline model](/tiflash/tiflash-pipeline-model.md). The default value is `OFF`, which means the pipeline model is disabled. |
| [`tidb_enable_tiflash_pipeline_model`](https://docs.pingcap.com/tidb/v7.2/system-variables#tidb_enable_tiflash_pipeline_model-new-in-v720) | Newly added | Controls whether to enable the new execution model of TiFlash, the [pipeline model](/tiflash/tiflash-pipeline-model.md). The default value is `OFF`, which means the pipeline model is disabled. |
| [`tidb_expensive_txn_time_threshold`](/system-variables.md#tidb_expensive_txn_time_threshold-new-in-v720) | Newly added | Controls the threshold for logging expensive transactions, which is 600 seconds by default. When the duration of a transaction exceeds the threshold, and the transaction is neither committed nor rolled back, it is considered an expensive transaction and will be logged. |

### Configuration file parameters
Expand Down
4 changes: 2 additions & 2 deletions sql-statements/sql-statement-calibrate-resource.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ This method mainly estimates capacity based on the current cluster configuration
- `OLTP_WRITE_ONLY`: applies to workloads with heavy data write. It is estimated based on a workload model similar to `sysbench oltp_write_only`.
- `OLTP_READ_WRITE`: applies to workloads with even data read and write. It is estimated based on a workload model similar to `sysbench oltp_read_write`.
- `OLTP_READ_ONLY`: applies to workloads with heavy data read. It is estimated based on a workload model similar to `sysbench oltp_read_only`.
- `TPCH_10`: AP queries for load forecasting based on 22 queries from `TPCH-10G`.
- `TPCH_10`: applies to AP queries. It is estimated based on 22 queries from `TPCH-10G`.

> **Note:**
>
Expand Down Expand Up @@ -100,7 +100,7 @@ CALIBRATE RESOURCE START_TIME '2023-04-18 08:00:00' DURATION '9m';
ERROR 1105 (HY000): the duration of calibration is too short, which could lead to inaccurate output. Please make the duration between 10m0s and 24h0m0s
```

The monitoring metrics for the [capacity estimation based on the actual workload](#estimate-capacity-based-on-actual-workload) feature include `tikv_cpu_quota`, `tidb_server_maxprocs`, `resource_manager_resource_unit`, `process_cpu_usage`, `tiflash_cpu_quota`, `tiflash_resource_manager_resource_unit` and `tiflash_process_cpu_usage`. If the CPU quota monitoring data is empty, there will be an error with the corresponding monitoring metric name, as shown in the following example:
The monitoring metrics for the [capacity estimation based on the actual workload](#estimate-capacity-based-on-actual-workload) feature include `tikv_cpu_quota`, `tidb_server_maxprocs`, `resource_manager_resource_unit`, `process_cpu_usage`, `tiflash_cpu_quota`, `tiflash_resource_manager_resource_unit`, and `tiflash_process_cpu_usage`. If the CPU quota monitoring data is empty, there will be an error with the corresponding monitoring metric name, as shown in the following example:

```sql
CALIBRATE RESOURCE START_TIME '2023-04-18 08:00:00' DURATION '60m';
Expand Down
20 changes: 0 additions & 20 deletions system-variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -2261,26 +2261,6 @@ Query OK, 0 rows affected (0.09 sec)

</CustomContent>

<CustomContent platform="tidb-cloud">

- Scope: SESSION | GLOBAL
- Persists to cluster: Yes
- Type: Boolean
- Default value: `OFF`
- This variable controls whether to enable the new [Pipeline Execution Model](https://docs.pingcap.com/tidb/v7.2/tiflash-pipeline-model) in TiFlash.
- When this variable is set to `OFF` to disable the pipeline execution model, the query pushed down to TiFlash will be executed using the original stream model.
- When this variable is set to `ON` to enable the pipeline execution model, the query pushed down to TiFlash will be executed using the new pipeline execution model.

> **Note:**
>
> - The pipeline execution model is currently an experimental feature and is not recommended to use in production environments.
> - The pipeline execution model does not support the following features. When the following features are enabled, even if `tidb_enable_tiflash_pipeline_model` is set to `ON`, the query pushed down to TiFlash will still be executed using the original stream model.
>
> - [Join operator spill to disk](#tidb_max_bytes_before_tiflash_external_join-new-in-v700)
> - [TiFlash Disaggregated Storage and Compute Architecture and S3 Support](https://docs.pingcap.com/tidb/v7.2/tiflash-disaggregated-and-s3)

</CustomContent>

### tidb_enable_tiflash_read_for_write_stmt <span class="version-mark">New in v6.3.0</span>

- Scope: SESSION | GLOBAL
Expand Down
31 changes: 12 additions & 19 deletions tidb-resource-control.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,10 @@ The TiDB resource control feature provides two layers of resource management cap

- TiKV scheduling: You can set the absolute priority [(`PRIORITY`)](/information-schema/information-schema-resource-groups.md#examples) as needed. Different resources are scheduled according to the `PRIORITY` setting. Tasks with high `PRIORITY` are scheduled first. If you do not set the absolute priority, TiKV uses the value of `RU_PER_SEC` of each resource group to determine the priority of the read and write requests for each resource group. Based on the priorities, the storage layer uses the priority queue to schedule and process requests.

Starting from v7.4.0, TiDB's resource management feature extends its support to control TiFlash resources. The principles behind TiFlash resource control and TiDB flow control and TiKV scheduling are similar:
Starting from v7.4.0, the resource control feature supports controlling TiFlash resources. Its principle is similar to that of TiDB flow control and TiKV scheduling:

- TiFlash Resource Control: Leveraging the [TiFlash Pipeline Model execution model](/tiflash/tiflash-pipeline-model.md), it can precisely monitor the CPU consumption for different queries and convert it into [Request Units (RU)](#What-is-Request-Unit-(RU)) for deduction. Traffic control is implemented using a token bucket algorithm.

- TiFlash Scheduling: When system resources become scarce, it schedules pipeline tasks among multiple resource groups based on their priorities. The specific logic involves assessing the resource group's priority `PRIORITY`, then considering CPU usage, and combining it with `RU_PER_SEC`. The final outcome is that if `rg1` and `rg2` have the same `PRIORITY` but `rg2` has double the `RU_PER_SEC` of `rg1`, then `rg2` can use twice the CPU time compared to `rg1`.
- TiFlash flow control: With the [TiFlash pipeline execution model](/tiflash/tiflash-pipeline-model.md), TiFlash can more accurately obtain the CPU consumption of different queries and convert it into [Request Units (RU)](#what-is-request-unit-ru) for deduction. Traffic control is implemented using a token bucket algorithm.
- TiFlash scheduling: When system resources are insufficient, TiFlash schedules pipeline tasks among multiple resource groups based on their priorities. The specific logic is: First, TiFlash assesses the `PRIORITY` of the resource group, then considers the CPU usage and `RU_PER_SEC`. As a result, if `rg1` and `rg2` have the same `PRIORITY` but the `RU_PER_SEC` of `rg2` is twice that of `rg1`, the CPU usage of `rg2` is twice that of `rg1`.

## Scenarios for resource control

Expand Down Expand Up @@ -86,7 +85,7 @@ Request Unit (RU) is a unified abstraction unit in TiDB for system resources, wh
</tbody>
</table>

Currently, TiFlash resource control only considers SQL CPU (i.e., the CPU time consumed by the execution of pipeline tasks for queries) and read request payload.
Currently, TiFlash resource control only considers SQL CPU, which is the CPU time consumed by the execution of pipeline tasks for queries, and read request payload.

> **Note:**
>
Expand All @@ -107,31 +106,27 @@ The resource control feature introduces the following system variables or parame
<CustomContent platform="tidb">

* TiKV: you can use the [`resource-control.enabled`](/tikv-configuration-file.md#resource-control) parameter to control whether to use request scheduling based on resource groups.
* TiFlash: you can use the [`tidb_enable_resource_control`](/system-variables.md#tidb_enable_resource_control-new-in-v660) system variable and the [`enable_resource_control`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) configuration item (introduced in v7.4.0) to control whether to enable TiFlash resource control.

</CustomContent>

<CustomContent platform="tidb-cloud">

* TiKV: For TiDB Self-Hosted, you can use the `resource-control.enabled` parameter to control whether to use request scheduling based on resource group quotas. For TiDB Cloud, the value of the `resource-control.enabled` parameter is `true` by default and does not support dynamic modification.
* TiFlash: For TiDB Self-Hosted, you can use the `tidb_enable_resource_control` system variable and the `enable_resource_control` configuration item (introduced in v7.4.0) to control whether to enable TiFlash resource control.

</CustomContent>

<CustomContent platform="tiflash">

* TiFlash: Control the enabling of TiFlash resource control by configuring the global variable [`tidb_enable_resource_control`](/system-variables.md#tidb_enable_resource_control-introduced-since-v660) and the TiFlash configuration option [`enable_resource_control`](/tiflash/tiflash-configuration.md#configuration-the-tiflashtoml-file) (introduced since v7.4.0).

</CustomContent>

Starting from TiDB v7.0.0, both parameters are enabled by default. The results of the combinations of these two parameters are shown in the following table.
Starting from TiDB v7.0.0, `tidb_enable_resource_control` and `resource-control.enabled` are enabled by default. The results of the combinations of these two parameters are shown in the following table.

| `resource-control.enabled` | `tidb_enable_resource_control`= ON | `tidb_enable_resource_control`= OFF |
|:----------------------------|:-------------------------------------|:-------------------------------------|
| `resource-control.enabled`= true | Flow control and scheduling (recommended) | Invalid combination |
| `resource-control.enabled`= false | Only flow control (not recommended) | The feature is disabled. |

Starting from v7.4.0, the TiFlash configuration option `enable_resource_control` is enabled by default, and it works in conjunction with `tidb_enable_resource_control` to control the TiFlash resource management feature. Only when both are enabled can TiFlash resource management perform flow control and priority scheduling. Additionally, when `enable_resource_control` is turned on, TiFlash utilizes the [Pipeline Model execution model](/tiflash/tiflash-pipeline-model.md).
Starting from v7.4.0, the TiFlash configuration item `enable_resource_control` is enabled by default. It works together with `tidb_enable_resource_control` to control the TiFlash resource control feature. TiFlash resource control only performs flow control and priority scheduling when both `enable_resource_control` and `tidb_enable_resource_control` are enabled. Additionally, when `enable_resource_control` is enabled, TiFlash uses the [Pipeline execution model](/tiflash/tiflash-pipeline-model.md).

For detailed information on the resource control implementation mechanism and related parameters, please refer to [RFC: Global Resource Control in TiDB](https://github.com/pingcap/tidb/blob/master/docs/design/2022-11-25-global-resource-control.md) and [TiFlash Resource Control](https://github.com/pingcap/tiflash/blob/master/docs/design/2023-09-21-tiflash-resource-control.md).
For more information about the resource control mechanism and parameters, see [RFC: Global Resource Control in TiDB](https://github.com/pingcap/tidb/blob/master/docs/design/2022-11-25-global-resource-control.md) and [TiFlash Resource Control](https://github.com/pingcap/tiflash/blob/master/docs/design/2023-09-21-tiflash-resource-control.md).

## How to use resource control

Expand Down Expand Up @@ -395,6 +390,8 @@ You can get more information about runaway queries from the following system tab

2. Set the TiKV parameter [`resource-control.enabled`](/tikv-configuration-file.md#resource-control) to `false` to disable scheduling based on the RU of the resource group.

3. Set the TiFlash configuration item [`enable_resource_control`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) to `false` to disable TiFlash resource control.

</CustomContent>

<CustomContent platform="tidb-cloud">
Expand All @@ -407,11 +404,7 @@ You can get more information about runaway queries from the following system tab

2. For TiDB Self-Hosted, you can use the `resource-control.enabled` parameter to control whether to use request scheduling based on resource group quotas. For TiDB Cloud, the value of the `resource-control.enabled` parameter is `true` by default and does not support dynamic modification. If you need to disable it for TiDB Dedicated clusters, contact [TiDB Cloud Support](/tidb-cloud/tidb-cloud-support.md).

</CustomContent>

<CustomContent platform="tidb-cloud">

To disable TiFlash resource control, set the TiFlash config [`enable_resource_control`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) to false.
3. For TiDB Self-Hosted, you can use the `enable_resource_control` configuration item to control whether to enable TiFlash resource control. For TiDB Cloud, <!--TODO: add the behavior of TiFlash resource control in TiDB Cloud-->.

</CustomContent>

Expand Down
2 changes: 1 addition & 1 deletion tiflash/tiflash-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -239,7 +239,7 @@ delta_index_cache_size = 0
## New in v7.0.0. This item specifies the maximum memory available for the HashJoin operator with EquiJoin before a disk spill is triggered. When the memory usage exceeds the threshold, HashJoin reduces memory usage by spilling to disk. This item defaults to 0, which means that the memory usage is unlimited and spill to disk is never used for HashJoin with EquiJoin.
max_bytes_before_external_join = 0

## Introduced starting from v7.4.0, it signifies whether to enable TiFlash resource management functionality. When set to true, TiFlash utilizes the Pipeline Model execution model.
## New in v7.4.0. This item controls whether to enable the TiFlash resource control feature. When it is set to true, TiFlash uses the pipeline execution model.
enable_resource_control = true

## Security settings take effect starting from v4.0.5.
Expand Down
7 changes: 3 additions & 4 deletions tiflash/tiflash-pipeline-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,10 @@ This document introduces the TiFlash pipeline execution model.

Starting from v7.2.0, TiFlash supports a new execution model, the pipeline execution model.

- **v7.2.0 and v7.3.0:** The TiFlash Pipeline Model was an experimental feature controlled by [`tidb_enable_tiflash_pipeline_model`](https://docs.pingcap.com/zh/tidb/v7.2/system-variables#tidb_enable_tiflash_pipeline_model-introduced-since-v720).
- For v7.2.0 and v7.3.0: The pipeline execution model is experimental and is controlled by [`tidb_enable_tiflash_pipeline_model`](https://docs.pingcap.com/tidb/v7.2/system-variables#tidb_enable_tiflash_pipeline_model-introduced-since-v720).
- For v7.4.0 and later versions: The pipeline execution model becomes generally available. It is an internal feature of TiFlash and is tightly integrated with TiFlash resource control. When you enable TiFlash resource control, the pipeline execution model is automatically enabled. For more information about how to use TiFlash resource control, refer to [Use resource control to achieve resource isolation](/tidb-resource-control.md#parameters-for-resource-control). Additionally, starting from v7.4.0, the system variable `tidb_enable_tiflash_pipeline_model` is deprecated.

- **v7.4.0 and later versions:** The Pipeline Model has become an official feature. It is an internal feature of TiFlash and is tightly integrated with TiFlash resource management. When you enable TiFlash resource management, the Pipeline Model is automatically enabled. For details on how to use TiFlash resource management, refer to [Using Resource Control to Achieve Resource Isolation](/tidb-resource-control.md#related-parameters). Additionally, starting from v7.4.0, the variable `tidb_enable_tiflash_pipeline_model` has been deprecated.

The Pipeline Model primarily draws inspiration from the paper titled [Morsel-Driven Parallelism: A NUMA-Aware Query Evaluation Framework for the Many-Core Age](https://dl.acm.org/doi/10.1145/2588555.2610507). It provides a sophisticated task scheduling model, distinct from traditional thread scheduling models. This approach reduces the overhead of operating system thread creation and scheduling while offering precise scheduling mechanisms.
Inspired by the paper [Morsel-Driven Parallelism: A NUMA-Aware Query Evaluation Framework for the Many-Core Age](https://dl.acm.org/doi/10.1145/2588555.2610507), the TiFlash pipeline execution model provides a fine-grained task scheduling model, which is different from the traditional thread scheduling model. It reduces the overhead of operating system thread scheduling and provides a fine-grained scheduling mechanism.

## Design and implementation

Expand Down