benchmark, TOC: add 4.0 TPC-H, TPC-C and Sysbench test reports (pingc…

…ap#2661) * add TPC-H benchmark test report * Add 4.0 tpcc,tpch and sysbench test reports * Update benchmark-sysbench-v4-vs-v3.md * address comments * remove s * address comment and add tpch result * correct headinglevel * Update v4.0-performance-benchmarking-with-tpcc.md * update tpcc image Co-authored-by: Caitin <34535727+CaitinChen@users.noreply.github.com> Co-authored-by: JaySon <tshent@qq.com>
Frank945946 · Jun 5, 2020 · f11484c · f11484c
1 parent e6dd2d8
commit f11484c
Show file tree

Hide file tree

Showing 10 changed files with 471 additions and 74 deletions.
diff --git a/TOC.md b/TOC.md
@@ -10,8 +10,10 @@
   + Benchmarks
     - [How to Test TiDB Using Sysbench](/benchmark/benchmark-tidb-using-sysbench.md)
     - [How to Run TPC-C Test on TiDB](/benchmark/benchmark-tidb-using-tpcc.md)
+    + [Sysbench Performance Test - v4.0 vs. v3.0](/benchmark/benchmark-sysbench-v4-vs-v3.md)
+    + [TPC-H Performance Test - v4.0 vs. v3.0](/benchmark/v4.0-performance-benchmarking-with-tpch.md)
+    + [TPC-C Performance Test - v4.0 vs. v3.0](/benchmark/v4.0-performance-benchmarking-with-tpcc.md)
     - [Sysbench Performance Test - v3.0 vs. v2.1](/benchmark/v3.0-performance-benchmarking-with-sysbench.md)
-    - [TPC-C Performance Test - v4.0 vs. v3.0](/benchmark/v4.0-performance-benchmarking-with-tpcc.md)
     - [TPC-C Performance Test - v3.0 vs. v2.1](/benchmark/v3.0-performance-benchmarking-with-tpcc.md)
     - [Interaction Test on Online Workloads and `ADD INDEX` Operations](/benchmark/online-workloads-and-add-index-operations.md)
 - [Quick Start with TiDB](/quick-start-with-tidb.md)

diff --git a/benchmark/benchmark-sysbench-v4-vs-v3.md b/benchmark/benchmark-sysbench-v4-vs-v3.md
@@ -0,0 +1,174 @@
+---
+title: TiDB Sysbench Performance Test Report -- v4.0 vs. v3.0
+summary: Compare the Sysbench performance of TiDB 4.0 and TiDB 3.0.
+category: benchmark
+---
+
+# TiDB Sysbench Performance Test Report -- v4.0 vs. v3.0
+
+## Test purpose
+
+This test aims to compare the Sysbench performance of TiDB 4.0 and TiDB 3.0 in the Online Transactional Processing (OLTP) scenario.
+
+## Test environment (AWS EC2）
+
+### Hardware configuration
+
+| Service type         | EC2 type     | Instance count |
+|:----------|:----------|:----------|
+| PD        | m5.xlarge |     3     |
+| TiKV      | i3.4xlarge|     3     |
+| TiDB      | c5.4xlarge|     3     |
+| Sysbench  | m5.4xlarge|     1     |
+
+### Software version
+
+| Service type   | Software version    |
+|:----------|:-----------|
+| PD        | 3.0 and 4.0   |
+| TiDB      | 3.0 and 4.0   |
+| TiKV      | 3.0 and 4.0   |
+| Sysbench  | 1.0.20     |
+
+### Parameter configuration
+
+#### TiDB v3.0 configuration
+
+{{< copyable "" >}}
+
+```yaml
+log.level: "error"
+performance.max-procs: 20
+prepared-plan-cache.enabled: true
+tikv-client.max-batch-wait-time: 2000000
+```
+
+#### TiKV v3.0 configuration
+
+{{< copyable "" >}}
+
+```yaml
+storage.scheduler-worker-pool-size: 5
+raftstore.store-pool-size: 3
+raftstore.apply-pool-size: 3
+rocksdb.max-background-jobs: 3
+raftdb.max-background-jobs: 3
+raftdb.allow-concurrent-memtable-write: true
+server.grpc-concurrency: 6
+readpool.storage.normal-concurrency: 10
+readpool.coprocessor.normal-concurrency: 5
+```
+
+#### TiDB v4.0 configuration
+
+{{< copyable "" >}}
+
+```yaml
+log.level: "error"
+performance.max-procs: 20
+prepared-plan-cache.enabled: true
+tikv-client.max-batch-wait-time: 2000000
+```
+
+#### TiKV v4.0 configuration
+
+{{< copyable "" >}}
+
+```yaml
+storage.scheduler-worker-pool-size: 5
+raftstore.store-pool-size: 3
+raftstore.apply-pool-size: 3
+rocksdb.max-background-jobs: 3
+raftdb.max-background-jobs: 3
+raftdb.allow-concurrent-memtable-write: true
+server.request-batch-enable-cross-command: false
+server.grpc-concurrency: 6
+readpool.unifiy-read-pool: true
+readpool.unified.min-thread-count: 5
+readpool.unified.max-thread-count: 20
+readpool.storage.normal-concurrency: 10
+pessimistic-txn.pipelined: true
+```
+
+#### Global variable configuration
+
+{{< copyable "sql" >}}
+
+```sql
+set global tidb_hashagg_final_concurrency=1;
+set global tidb_hashagg_partial_concurrency=1;
+set global tidb_disable_txn_auto_retry=0;
+```
+
+## Test plan
+
+1. Deploy TiDB v4.0 and v3.0 using TiUP.
+2. Use Sysbench to import 16 tables, each table with 10 million rows of data.
+3. Execute the `analyze table` statement on each table.
+4. Back up the data used for restore before different concurrency tests, which ensures data consistency for each test.
+5. Start the Sysbench client to perform the `point_select`, `read_write`, `update_index`, and `update_non_index` tests. Perform stress tests on TiDB via AWS NLB. In each type of test, the warm-up takes 1 minute and the test takes 5 minutes.
+6. After each type of test is completed, stop the cluster, overwrite the cluster with the backup data in step 4, and restart the cluster.
+
+### Prepare test data
+
+Execute the following command to prepare the test data:
+
+{{< copyable "shell-regular" >}}
+
+```bash
+sysbench oltp_common \
+    --threads=16 \
+    --rand-type=uniform \
+    --db-driver=mysql \
+    --mysql-db=sbtest \
+    --mysql-host=$aws_nlb_host \
+    --mysql-port=$aws_nlb_port \
+    --mysql-user=root \
+    --mysql-password=password \
+    prepare --tables=16 --table-size=10000000
+```
+
+### Perform the test
+
+Execute the following command to perform the test.
+
+{{< copyable "shell-regular" >}}
+
+```bash
+sysbench $testname \
+    --threads=$threads \
+    --time=300 \
+    --report-interval=1 \
+    --rand-type=uniform \
+    --db-driver=mysql \
+    --mysql-db=sbtest \
+    --mysql-host=$aws_nlb_host \
+    --mysql-port=$aws_nlb_port \
+    run --tables=16 --table-size=10000000
+```
+
+## Test results
+
+### Point Select performance
+
+Compared with v3.0, the Point Select performance of TiDB v4.0 has increased by 14%.
+
+![Point Select](/media/sysbench-v4vsv3-point-select.png)
+
+### Update Non-index performance
+
+Compared with v3.0, the Update Non-index performance of TiDB v4.0 has increased by 15%.
+
+![Update Non-index](/media/sysbench-v4vsv3-update-non-index.png)
+
+### Update Index performance
+
+Compared with v3.0, the Update Index performance of TiDB v4.0 has increased by 17%.
+
+![Update Index](/media/sysbench-v4vsv3-update-index.png)
+
+### Read-write performance
+
+Compared with v3.0, the read-write performance of TiDB v4.0 has increased by 31%.
+
+![Read Write](/media/sysbench-v4vsv3-read-write.png)
diff --git a/benchmark/v4.0-performance-benchmarking-with-tpcc.md b/benchmark/v4.0-performance-benchmarking-with-tpcc.md
@@ -1,113 +1,133 @@
 ---
 title: TiDB TPC-C Performance Test Report -- v4.0 vs. v3.0
-summary: Compares the TPC-C performance of TiDB 4.0 and TiDB 3.0 using BenchmarkSQL.
+summary: Compare the TPC-C performance of TiDB 4.0 and TiDB 3.0 using BenchmarkSQL.
 category: benchmark
-aliases: ['/docs/dev/benchmark/tpcc-v2/']
 ---
 
 # TiDB TPC-C Performance Test Report -- v4.0 vs. v3.0
 
 ## Test purpose
 
-This test aims to compare the TPC-C performance of TiDB 4.0 and TiDB 3.0.
+This test aims to compare the TPC-C performance of TiDB 4.0 and TiDB 3.0 in the Online Transactional Processing (OLTP) scenario.
 
-## Test version, time, and place
+## Test environment (AWS EC2)
 
-TiDB versions: v4.0.0-rc vs. v3.0.8
+### Hardware configuration
 
-Time: April, 2020
+| Service type   | EC2 type   |    Instance count  |
+|:----------|:----------|:----------|
+| PD        | m5.xlarge |     3     |
+| TiKV      | i3.4xlarge|     3     |
+| TiDB      | c5.4xlarge|     3     |
+| Sysbench  | m5.4xlarge|     1     |
 
-Place: Beijing
+### Software version
 
-## Test environment
+| Service type   | Software version    |
+|:----------|:-----------|
+| PD        | 3.0 and 4.0   |
+| TiDB      | 3.0 and 4.0   |
+| TiKV      | 3.0 and 4.0   |
+| BenchmarkSQL  | None     |
 
-AWS machine:
+### Parameter configuration
 
-| Type | Machine | Number | vCPUs |
-| :- | :- | :- | :- |
-| TiKV | i3.4xlarge | 3 | 16 |
-| TiDB | c5.4xlarge | 2 | 16 |
-| PD | m5.xlarge | 3 | 4 |
+#### TiDB v3.0 configuration
 
-This test uses the open-source BenchmarkSQL 5.0 as the TPC-C testing tool and adds the support for the MySQL protocol. You can download the testing program by using the following command:
-
-{{< copyable "shell-regular" >}}
+{{< copyable "" >}}
 
-```shell
-git clone -b 5.0-mysql-support-opt https://github.com/pingcap/benchmarksql.git
+```yaml
+log.level: "error"
+performance.max-procs: 20
+prepared-plan-cache.enabled: true
+tikv-client.max-batch-wait-time: 2000000
 ```
 
-## Test Plan
-
-Use BenchmarkSQL to load the data of **5000 warehouses** into the TiDB cluster. By using HAProxy, send concurrent requests to the cluster at an incremental number. A single concurrent test lasts 10 minutes.
+#### TiKV v3.0 configuration
 
-### TiDB version information
+{{< copyable "" >}}
 
-#### v4.0.0-rc
+```yaml
+storage.scheduler-worker-pool-size: 5
+raftstore.store-pool-size: 3
+raftstore.apply-pool-size: 3
+rocksdb.max-background-jobs: 3
+raftdb.max-background-jobs: 3
+raftdb.allow-concurrent-memtable-write: true
+server.grpc-concurrency: 6
+readpool.storage.normal-concurrency: 10
+readpool.coprocessor.normal-concurrency: 5
+```
 
-| Component | GitHash |
-| :- | :- |
-| TiDB | 79db9e30ab8f98ac07c8ae55c66dfecc24b43d56 |
-| TiKV | f45d0c963df3ee4b1011caf5eb146cacd1fbbad8 |
-| PD | 6f06805f3b0070107fcb4af68b2fc224dee0714d |
+#### TiDB v4.0 configuration
 
-#### v3.0.8
+{{< copyable "" >}}
 
-| Component | GitHash |
-| :- | :- |
-| TiDB | 8f13cf1449bd8903ff465a4f12ed89ecbac858a4 |
-| TiKV | 0d3168cfd8224fbc48a07796df83ddac0fbcbf46 |
-| PD | 456c42b8b0955b33426b58054e43b771801a74d0 |
+```yaml
+log.level: "error"
+performance.max-procs: 20
+prepared-plan-cache.enabled: true
+tikv-client.max-batch-wait-time: 2000000
+```
 
-### TiDB parameter configuration
+#### TiKV v4.0 configuration
 
 {{< copyable "" >}}
 
-```toml
-[log]
-level = "error"
-[performance]
-max-procs = 20
-[prepared_plan_cache]
-enabled = true
+```yaml
+storage.scheduler-worker-pool-size: 5
+raftstore.store-pool-size: 3
+raftstore.apply-pool-size: 3
+rocksdb.max-background-jobs: 3
+raftdb.max-background-jobs: 3
+raftdb.allow-concurrent-memtable-write: true
+server.request-batch-enable-cross-command: false
+server.grpc-concurrency: 6
+readpool.unifiy-read-pool: true
+readpool.unified.min-thread-count: 5
+readpool.unified.max-thread-count: 20
+readpool.storage.normal-concurrency: 10
+pessimistic-txn.pipelined: true
 ```
 
-### TiKV parameter configuration
+#### Global variable configuration
 
-{{< copyable "" >}}
+{{< copyable "sql" >}}
 
-```toml
-[readpool]
-unify-read-pool = true
-[readpool.unified]
-min-thread-count = 5
-max-thread-count = 6
-[readpool.storage]
-[readpool.coprocessor]
-[storage]
-scheduler-worker-pool-size = 3
-[raftstore]
-store-pool-size = 3
-[rocksdb]
-max-background-jobs = 3
-wal-dir = ""
-[raftdb]
-max-background-jobs = 3
-allow-concurrent-memtable-write = true
-[server]
-request-batch-enable-cross-command = false
-[pessimistic-txn]
-pipelined = true
+```sql
+set global tidb_hashagg_final_concurrency=1;
+set global tidb_hashagg_partial_concurrency=1;
+set global tidb_disable_txn_auto_retry=0;
 ```
 
-### Cluster topology
+## Test plan
+
+1. Deploy TiDB v4.0 and v3.0 using TiUP.
+
+2. Use BenchmarkSQL to import the TPC-C 5000 Warehouse data.
+
+    1. Compile BenchmarkSQL：
+
+        {{< copyable "bash" >}}
+
+        ```bash
+        git clone https://github.com/pingcap/benchmarksql && cd benchmarksql && ant
+        ```
+
+    2. Enter the `run` directory, edit the `props.mysql` file according to the actual situation, and modify the `conn`, `warehouses`, `loadWorkers`, `terminals`, and `runMins` configuration items.
+
+    3. Execute the `runSQL.sh ./props.mysql sql.mysql/tableCreates.sql` command.
+
+    4. Execute the `runSQL.sh ./props.mysql sql.mysql/indexCreates.sql` command.
+
+    5. Run MySQL client and execute the `analyze table` statement on every table.
+
+3. Execute the `runBenchmark.sh ./props.mysql` command.
 
-* 1 TiKV instance is deployed on each `i3.4xlarge` machine, 3 in total.
-* 1 TiDB instance is deployed on each `c5.4xlarge` machine, 2 in total.
-* 1 PD instance is deployed on each `m5.xlarge` machine, 3 in total.
+4. Extract the tpmC data of New Order from the result.
 
 ## Test result
 
-![tpcc](/media/tpcc-3.0-4.0.png)
+According to the test statistics, the TPC-C performance of TiDB v4.0 has **increased by 50%** compared with that of TiDB v3.0.
 
-According to the testing statistics, the performance of TiDB 4.0.0-rc **has increased by 45%** than that of TiDB 3.0.8.
+![TPC-C](/media/tpcc-v4vsv3.png)