benchmark, TOC: add 4.0 TPC-H, TPC-C and Sysbench test reports #2661
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,174 @@ | ||
| --- | ||
| title: TiDB Sysbench Performance Test Report -- v4.0 vs. v3.0 | ||
| summary: Compare the Sysbench performance of TiDB 4.0 and TiDB 3.0. | ||
| category: benchmark | ||
| --- | ||
|
|
||
| # TiDB Sysbench Performance Test Report -- v4.0 vs. v3.0 | ||
|
|
||
| ## Test purpose | ||
|
|
||
| This test aims to compare the Sysbench performances of TiDB 4.0 and TiDB 3.0 in the Online Transactional Processing (OLTP) scenario. | ||
|
|
||
| ## Test environment (AWS EC2） | ||
|
|
||
| ### Hardware configuration | ||
|
|
||
| | Service type | EC2 type | Instance count | | ||
| |:----------|:----------|:----------| | ||
| | PD | m5.xlarge | 3 | | ||
| | TiKV | i3.4xlarge| 3 | | ||
| | TiDB | c5.4xlarge| 3 | | ||
| | Sysbench | m5.4xlarge| 1 | | ||
|
|
||
| ### Software version | ||
|
|
||
| | Service type | Software version | | ||
| |:----------|:-----------| | ||
| | PD | 3.0 and 4.0 | | ||
| | TiDB | 3.0 and 4.0 | | ||
| | TiKV | 3.0 and 4.0 | | ||
| | Sysbench | 1.0.20 | | ||
|
|
||
| ### Parameter configuration | ||
|
|
||
| #### TiDB v3.0 configuration | ||
|
|
||
| {{< copyable "" >}} | ||
|
|
||
| ```yaml | ||
| log.level: "error" | ||
| performance.max-procs: 20 | ||
| prepared-plan-cache.enabled: true | ||
| tikv-client.max-batch-wait-time: 2000000 | ||
| ``` | ||
|
|
||
| #### TiKV v3.0 configuration | ||
|
|
||
| {{< copyable "" >}} | ||
|
|
||
| ```yaml | ||
| storage.scheduler-worker-pool-size: 5 | ||
| raftstore.store-pool-size: 3 | ||
| raftstore.apply-pool-size: 3 | ||
| rocksdb.max-background-jobs: 3 | ||
| raftdb.max-background-jobs: 3 | ||
| raftdb.allow-concurrent-memtable-write: true | ||
| server.grpc-concurrency: 6 | ||
| readpool.storage.normal-concurrency: 10 | ||
| readpool.coprocessor.normal-concurrency: 5 | ||
| ``` | ||
|
|
||
| #### TiDB v4.0 configuration | ||
|
|
||
| {{< copyable "" >}} | ||
|
|
||
| ```yaml | ||
| log.level: "error" | ||
| performance.max-procs: 20 | ||
| prepared-plan-cache.enabled: true | ||
| tikv-client.max-batch-wait-time: 2000000 | ||
| ``` | ||
|
|
||
| #### TiKV v4.0 configuration | ||
|
|
||
| {{< copyable "" >}} | ||
|
|
||
| ```yaml | ||
| storage.scheduler-worker-pool-size: 5 | ||
| raftstore.store-pool-size: 3 | ||
| raftstore.apply-pool-size: 3 | ||
| rocksdb.max-background-jobs: 3 | ||
| raftdb.max-background-jobs: 3 | ||
| raftdb.allow-concurrent-memtable-write: true | ||
| server.request-batch-enable-cross-command: false | ||
| server.grpc-concurrency: 6 | ||
| readpool.unifiy-read-pool: true | ||
| readpool.unified.min-thread-count: 5 | ||
| readpool.unified.max-thread-count: 20 | ||
| readpool.storage.normal-concurrency: 10 | ||
| pessimistic-txn.pipelined: true | ||
| ``` | ||
|
|
||
| #### Global variable configuration | ||
|
|
||
| {{< copyable "sql" >}} | ||
|
|
||
| ```sql | ||
| set global tidb_hashagg_final_concurrency=1; | ||
| set global tidb_hashagg_partial_concurrency=1; | ||
| set global tidb_disable_txn_auto_retry=0; | ||
| ``` | ||
|
|
||
| ## Test plan | ||
|
|
||
| 1. Deploy TiDB v4.0 and v3.0 using TiUP. | ||
| 2. Use Sysbench to import 16 tables, each table with 10 million rows of data. | ||
| 3. Execute the `analyze table` statement on each table. | ||
| 4. Back up the data used for restore before different concurrency tests, which ensures data consistency for each test. | ||
| 5. Start the Sysbench client to perform the `point_select`, `read_write`, `update_index`, and `update_non_index` tests. Pressurize TiDB via AWS NLB. In each type of test, the warm-up takes 1 minute and the test takes 5 minutes. | ||
| 6. After each type of test is completed, stop the cluster, overwrite the cluster with the backup data in step 4, and restart the cluster. | ||
|
|
||
| ### Prepare test data | ||
|
|
||
| Execute the following command to prepare the test data: | ||
|
|
||
| {{< copyable "shell-regular" >}} | ||
|
|
||
| ```bash | ||
| sysbench oltp_common \ | ||
| --threads=16 \ | ||
| --rand-type=uniform \ | ||
| --db-driver=mysql \ | ||
| --mysql-db=sbtest \ | ||
| --mysql-host=$aws_nlb_host \ | ||
| --mysql-port=$aws_nlb_port \ | ||
| --mysql-user=root \ | ||
| --mysql-password=password \ | ||
| prepare --tables=16 --table-size=10000000 | ||
| ``` | ||
|
|
||
| ### Perform the test | ||
|
|
||
| Execute the following command to perform the test. | ||
|
|
||
| {{< copyable "shell-regular" >}} | ||
|
|
||
| ```bash | ||
| sysbench $testname \ | ||
| --threads=$threads \ | ||
| --time=300 \ | ||
| --report-interval=1 \ | ||
| --rand-type=uniform \ | ||
| --db-driver=mysql \ | ||
| --mysql-db=sbtest \ | ||
| --mysql-host=$aws_nlb_host \ | ||
| --mysql-port=$aws_nlb_port \ | ||
| run --tables=16 --table-size=10000000 | ||
| ``` | ||
|
|
||
| ## Test results | ||
|
|
||
| ### Point Select performance | ||
|
|
||
| Compared with v3.0, the Point Select performance of TiDB v4.0 has increased by 14%. | ||
|
|
||
|  | ||
|
|
||
| ### Update Non-index performance | ||
|
|
||
| Compared with v3.0, the Update Non-index performance of TiDB v4.0 has increased by 15%. | ||
|
|
||
|  | ||
|
|
||
| ### Update Index performance | ||
|
|
||
| Compared with v3.0, the Update Index performance of TiDB v4.0 has increased by 17%. | ||
|
|
||
|  | ||
|
|
||
| ### Read-write performance | ||
|
|
||
| Compared with v3.0, the read-write performance of TiDB v4.0 has increased by 31%. | ||
|
|
||
|  | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -1,113 +1,133 @@ | ||||||
| --- | ||||||
| title: TiDB TPC-C Performance Test Report -- v4.0 vs. v3.0 | ||||||
| summary: Compare the TPC-C performance of TiDB 4.0 and TiDB 3.0 using BenchmarkSQL. | ||||||
| category: benchmark | ||||||
| --- | ||||||
|
|
||||||
| # TiDB TPC-C Performance Test Report -- v4.0 vs. v3.0 | ||||||
|
|
||||||
| ## Test purpose | ||||||
|
|
||||||
| This test aims to compare the TPC-C performances of TiDB 4.0 and TiDB 3.0 in the Online Transactional Processing (OLTP) scenario. | ||||||
|
|
||||||
| ## Test environment (AWS EC2) | ||||||
|
|
||||||
| ### Hardware configuration | ||||||
|
|
||||||
| | Service type | EC2 type | Instance count | | ||||||
| |:----------|:----------|:----------| | ||||||
| | PD | m5.xlarge | 3 | | ||||||
| | TiKV | i3.4xlarge| 3 | | ||||||
| | TiDB | c5.4xlarge| 3 | | ||||||
| | Sysbench | m5.4xlarge| 1 | | ||||||
|
|
||||||
| ### Software version | ||||||
|
|
||||||
| | Service type | Software version | | ||||||
| |:----------|:-----------| | ||||||
| | PD | 3.0 and 4.0 | | ||||||
| | TiDB | 3.0 and 4.0 | | ||||||
| | TiKV | 3.0 and 4.0 | | ||||||
| | BenchmarkSQL | None | | ||||||
|
|
||||||
| ### Parameter configuration | ||||||
|
|
||||||
| #### TiDB v3.0 configuration | ||||||
|
|
||||||
| {{< copyable "" >}} | ||||||
|
|
||||||
| ```yaml | ||||||
| log.level: "error" | ||||||
| performance.max-procs: 20 | ||||||
| prepared-plan-cache.enabled: true | ||||||
| tikv-client.max-batch-wait-time: 2000000 | ||||||
| ``` | ||||||
|
|
||||||
| #### TiKV v3.0 configuration | ||||||
|
|
||||||
| {{< copyable "" >}} | ||||||
|
|
||||||
| ```yaml | ||||||
| storage.scheduler-worker-pool-size: 5 | ||||||
| raftstore.store-pool-size: 3 | ||||||
| raftstore.apply-pool-size: 3 | ||||||
| rocksdb.max-background-jobs: 3 | ||||||
| raftdb.max-background-jobs: 3 | ||||||
| raftdb.allow-concurrent-memtable-write: true | ||||||
| server.grpc-concurrency: 6 | ||||||
| readpool.storage.normal-concurrency: 10 | ||||||
| readpool.coprocessor.normal-concurrency: 5 | ||||||
| ``` | ||||||
|
|
||||||
| #### TiDB v4.0 configuration | ||||||
|
|
||||||
| {{< copyable "" >}} | ||||||
|
|
||||||
| ```yaml | ||||||
| log.level: "error" | ||||||
| performance.max-procs: 20 | ||||||
| prepared-plan-cache.enabled: true | ||||||
| tikv-client.max-batch-wait-time: 2000000 | ||||||
| ``` | ||||||
|
|
||||||
| #### TiKV v4.0 configuration | ||||||
|
|
||||||
| {{< copyable "" >}} | ||||||
|
|
||||||
| ```yaml | ||||||
| storage.scheduler-worker-pool-size: 5 | ||||||
| raftstore.store-pool-size: 3 | ||||||
| raftstore.apply-pool-size: 3 | ||||||
| rocksdb.max-background-jobs: 3 | ||||||
| raftdb.max-background-jobs: 3 | ||||||
| raftdb.allow-concurrent-memtable-write: true | ||||||
| server.request-batch-enable-cross-command: false | ||||||
| server.grpc-concurrency: 6 | ||||||
| readpool.unifiy-read-pool: true | ||||||
| readpool.unified.min-thread-count: 5 | ||||||
| readpool.unified.max-thread-count: 20 | ||||||
| readpool.storage.normal-concurrency: 10 | ||||||
| pessimistic-txn.pipelined: true | ||||||
| ``` | ||||||
|
|
||||||
| #### Global variable configuration | ||||||
|
|
||||||
| {{< copyable "sql" >}} | ||||||
|
|
||||||
| ```sql | ||||||
| set global tidb_hashagg_final_concurrency=1; | ||||||
| set global tidb_hashagg_partial_concurrency=1; | ||||||
| set global tidb_disable_txn_auto_retry=0; | ||||||
| ``` | ||||||
|
|
||||||
| ### Test plan | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| 1. Deploy TiDB v4.0 and v3.0 using TiUP. | ||||||
|
|
||||||
| 2. Use BenchmarkSQL to import the TPC-C 5000 Warehouse data. | ||||||
|
|
||||||
| 1. Compile BenchmarkSQL： | ||||||
|
|
||||||
| {{< copyable "bash" >}} | ||||||
|
|
||||||
| ```bash | ||||||
| git clone https://github.com/pingcap/benchmarksql && cd benchmarksql && ant | ||||||
| ``` | ||||||
|
|
||||||
| 2. Enter the `run` directory, edit the `props.mysql` file according to the actual situation, and modify the `conn`, `warehouses`, `loadWorkers`, `terminals`, and `runMins` configuration items. | ||||||
|
|
||||||
| 3. Execute the `runSQL.sh ./props.mysql sql.mysql/tableCreates.sql` command. | ||||||
|
|
||||||
| 4. Execute the `runSQL.sh ./props.mysql sql.mysql/indexCreates.sql` command. | ||||||
|
|
||||||
| 5. Run MySQL client and execute the `analyze table` statement on every table. | ||||||
|
|
||||||
| 3. Execute the `runBenchmark.sh ./props.mysql` command. | ||||||
|
|
||||||
| 4. Extract the tpmC data of New Order from the result. | ||||||
|
|
||||||
| ### Test result | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| According to the test statistics, the TPC-C performance of TiDB v4.0 has **increased by 50%** compared with that of TiDB v3.0. | ||||||
|
|
||||||
|  | ||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@scsldb Please check the translation for "单轮预热 1 分钟，测试 5 分钟"