Added the EN version of the Benchmark reports (pingcap#5255)

Frank945946 · Apr 7, 2021 · f87307f · f87307f
1 parent 1d14a7c
commit f87307f
Show file tree

Hide file tree

Showing 8 changed files with 374 additions and 1 deletion.
diff --git a/TOC.md b/TOC.md
@@ -11,8 +11,9 @@
   + [Basic Features](/basic-features.md)
   + [Experimental Features](/experimental-features.md)
   + Benchmarks
+    + [v5.0 Sysbench Performance Test Report](/benchmark/benchmark-sysbench-v5-vs-v4.md)
+    + [v5.0 TPC-C Performance Test Report](/benchmark/v5.0-performance-benchmarking-with-tpcc.md)
     + [v5.0 MPP mode TPC-H 100GB Performance Test](/benchmark/v5.0-performance-benchmarking-with-tpch.md)
-    + [Interaction Test on Online Workloads and `ADD INDEX`](/benchmark/online-workloads-and-add-index-operations.md)
   + [MySQL Compatibility](/mysql-compatibility.md)
   + [TiDB Limitations](/tidb-limitations.md)
   + [TiDB Adopters](/adopters.md)

diff --git a/benchmark/benchmark-sysbench-v5-vs-v4.md b/benchmark/benchmark-sysbench-v5-vs-v4.md
@@ -0,0 +1,223 @@
+---
+title: TiDB Sysbench Performance Test Report -- v5.0 vs. v4.0
+---
+
+# TiDB Sysbench Performance Test Report -- v5.0 vs. v4.0
+
+## Test purpose
+
+This test aims at comparing the Sysbench performance of TiDB v5.0 and TiDB v4.0 in the Online Transactional Processing (OLTP) scenario.
+
+## Test environment (AWS EC2）
+
+### Hardware configuration
+
+| Service type         | EC2 type     | Instance count |
+|:----------|:----------|:----------|
+| PD        | m5.xlarge |     3     |
+| TiKV      | i3.4xlarge|     3     |
+| TiDB      | c5.4xlarge|     3     |
+| Sysbench  | c5.9xlarge|     1     |
+
+### Software version
+
+| Service type   | Software version    |
+|:----------|:-----------|
+| PD        | 4.0 and 5.0   |
+| TiDB      | 4.0 and 5.0   |
+| TiKV      | 4.0 and 5.0   |
+| Sysbench  | 1.0.20     |
+
+### Parameter configuration
+
+#### TiDB v4.0 configuration
+
+{{< copyable "" >}}
+
+```yaml
+log.level: "error"
+performance.max-procs: 20
+prepared-plan-cache.enabled: true
+tikv-client.max-batch-wait-time: 2000000
+```
+
+#### TiKV v4.0 configuration
+
+{{< copyable "" >}}
+
+```yaml
+storage.scheduler-worker-pool-size: 5
+raftstore.store-pool-size: 3
+raftstore.apply-pool-size: 3
+rocksdb.max-background-jobs: 3
+raftdb.max-background-jobs: 3
+raftdb.allow-concurrent-memtable-write: true
+server.request-batch-enable-cross-command: false
+server.grpc-concurrency: 6
+readpool.unified.min-thread-count: 5
+readpool.unified.max-thread-count: 20
+readpool.storage.normal-concurrency: 10
+pessimistic-txn.pipelined: true
+```
+
+#### TiDB v5.0 configuration
+
+{{< copyable "" >}}
+
+```yaml
+log.level: "error"
+performance.max-procs: 20
+prepared-plan-cache.enabled: true
+tikv-client.max-batch-wait-time: 2000000
+```
+
+#### TiKV v5.0 configuration
+
+{{< copyable "" >}}
+
+```yaml
+storage.scheduler-worker-pool-size: 5
+raftstore.store-pool-size: 3
+raftstore.apply-pool-size: 3
+rocksdb.max-background-jobs: 8
+raftdb.max-background-jobs: 4
+raftdb.allow-concurrent-memtable-write: true
+server.request-batch-enable-cross-command: false
+server.grpc-concurrency: 6
+readpool.unified.min-thread-count: 5
+readpool.unified.max-thread-count: 20
+readpool.storage.normal-concurrency: 10
+pessimistic-txn.pipelined: true
+```
+
+#### TiDB v4.0 global variable configuration
+
+{{< copyable "sql" >}}
+
+```sql
+set global tidb_hashagg_final_concurrency=1;
+set global tidb_hashagg_partial_concurrency=1;
+```
+
+#### TiDB v5.0 global variable configuration
+
+{{< copyable "sql" >}}
+
+```sql
+set global tidb_hashagg_final_concurrency=1;
+set global tidb_hashagg_partial_concurrency=1;
+set global tidb_enable_async_commit = 1;
+set global tidb_enable_1pc = 1;
+set global tidb_guarantee_linearizability = 0;
+set global tidb_enable_clustered_index = 1; 
+
+```
+
+## Test plan
+
+1. Deploy TiDB v5.0 and v4.0 using TiUP.
+2. Use Sysbench to import 16 tables, each table with 10 million rows of data.
+3. Execute the `analyze table` statement on each table.
+4. Back up the data used for restore before different concurrency tests, which ensures data consistency for each test.
+5. Start the Sysbench client to perform the `point_select`, `read_write`, `update_index`, and `update_non_index` tests. Perform stress tests on TiDB via AWS NLB. In each type of test, the warm-up takes 1 minute and the test takes 5 minutes.
+6. After each type of test is completed, stop the cluster, overwrite the cluster with the backup data in step 4, and restart the cluster.
+
+### Prepare test data
+
+Execute the following command to prepare the test data:
+
+{{< copyable "shell-regular" >}}
+
+```bash
+sysbench oltp_common \
+    --threads=16 \
+    --rand-type=uniform \
+    --db-driver=mysql \
+    --mysql-db=sbtest \
+    --mysql-host=$aws_nlb_host \
+    --mysql-port=$aws_nlb_port \
+    --mysql-user=root \
+    --mysql-password=password \
+    prepare --tables=16 --table-size=10000000
+```
+
+### Perform the test
+
+Execute the following command to perform the test.
+
+{{< copyable "shell-regular" >}}
+
+```bash
+sysbench $testname \
+    --threads=$threads \
+    --time=300 \
+    --report-interval=1 \
+    --rand-type=uniform \
+    --db-driver=mysql \
+    --mysql-db=sbtest \
+    --mysql-host=$aws_nlb_host \
+    --mysql-port=$aws_nlb_port \
+    run --tables=16 --table-size=10000000
+```
+
+## Test results
+
+### Point Select performance
+
+| Threads   | v4.0 QPS   | v4.0 95% latency (ms)   | v5.0 QPS   | v5.0 95% latency (ms)   | QPS improvement   |
+|:----------|:----------|:----------|:----------|:----------|:----------|
+| 150        | 159451.19        | 1.32        | 177876.25        | 1.23        | 11.56%     |
+| 300        | 244790.38        | 1.96        | 252675.03        | 1.82        | 3.22%     |
+| 600        | 322929.05        | 3.75        | 331956.84        | 3.36        | 2.80%     |
+| 900        | 364840.05        | 5.67        | 365655.04        | 5.09        | 0.22%     |
+| 1200        | 376529.18        | 7.98        | 366507.47        | 7.04        | -2.66%     |
+| 1500        | 368390.52        | 10.84        | 372476.35        | 8.90        | 1.11%     |
+
+Compared with v4.0, the Point Select performance of TiDB v5.0 has increased by 2.7%.
+
+![Point Select](/media/sysbench_v5vsv4_point_select.png)
+
+### Update Non-index performance
+
+| Threads   | v4.0 QPS   | v4.0 95% latency (ms)   | v5.0 QPS   | v5.0 95% latency (ms)   | QPS improvement   |
+|:----------|:----------|:----------|:----------|:----------|:----------|
+| 150        | 17243.78        | 11.04        | 30866.23        | 6.91        | 79.00%     |
+| 300        | 25397.06        | 15.83        | 45915.39        | 9.73        | 80.79%     |
+| 600        | 33388.08        | 25.28        | 60098.52        | 16.41        | 80.00%     |
+| 900        | 38291.75        | 36.89        | 70317.41        | 21.89        | 83.64%     |
+| 1200        | 41003.46        | 55.82        | 76376.22        | 28.67        | 86.27%     |
+| 1500        | 44702.84        | 62.19        | 80234.58        | 34.95        | 79.48%     |
+
+Compared with v4.0, the Update Non-index performance of TiDB v5.0 has increased by 81%.
+
+![Update Non-index](/media/sysbench_v5vsv4_update_non_index.png)
+
+### Update Index performance
+
+| Threads   | v4.0 QPS   | v4.0 95% latency (ms)   | v5.0 QPS   | v5.0 95% latency (ms)   | QPS improvement   |
+|:----------|:----------|:----------|:----------|:----------|:----------|
+| 150        | 11736.21        | 17.01        | 15631.34        | 17.01        | 33.19%     |
+| 300        | 15435.95        | 28.67        | 19957.06        | 22.69        | 29.29%     |
+| 600        | 18983.21        | 49.21        | 23218.14        | 41.85        | 22.31%     |
+| 900        | 20855.29        | 74.46        | 26226.76        | 53.85        | 25.76%     |
+| 1200        | 21887.64        | 102.97        | 28505.41        | 69.29        | 30.24%     |
+| 1500        | 23621.15        | 110.66        | 30341.06        | 82.96        | 28.45%     |
+
+Compared with v4.0, the Update Index performance of TiDB v5.0 has increased by 28%.
+
+![Update Index](/media/sysbench_v5vsv4_update_index.png)
+
+### Read Write performance
+
+| Threads   | v4.0 QPS   | v4.0 95% latency (ms)   | v5.0 QPS   | v5.0 95% latency (ms)   | QPS improvement   |
+|:----------|:----------|:----------|:----------|:----------|:----------|
+| 150        | 59979.91        | 61.08        | 66098.57        | 55.82        | 10.20%     |
+| 300        | 77118.32        | 102.97        | 84639.48        | 90.78        | 9.75%     |
+| 600        | 90619.52        | 183.21        | 101477.46        | 167.44        | 11.98%     |
+| 900        | 97085.57        | 267.41        | 109463.46        | 240.02        | 12.75%     |
+| 1200        | 106521.61        | 331.91        | 115416.05        | 320.17        | 8.35%     |
+| 1500        | 116278.96        | 363.18        | 118807.5        | 411.96        | 2.17%     |
+
+Compared with v4.0, the read-write performance of TiDB v5.0 has increased by 9%.
+
+![Read Write](/media/sysbench_v5vsv4_read_write.png)
diff --git a/benchmark/v5.0-performance-benchmarking-with-tpcc.md b/benchmark/v5.0-performance-benchmarking-with-tpcc.md
@@ -0,0 +1,149 @@
+---
+title: TiDB TPC-C Performance Test Report -- v5.0 vs. v4.0
+---
+
+# TiDB TPC-C Performance Test Report -- v5.0 vs. v4.0
+
+## Test purpose
+
+This test aims at comparing the TPC-C performance of TiDB v5.0 and TiDB v4.0 in the Online Transactional Processing (OLTP) scenario.
+
+## Test environment (AWS EC2)
+
+### Hardware configuration
+
+| Service type   | EC2 type   |    Instance count  |
+|:----------|:----------|:----------|
+| PD        | m5.xlarge |     3     |
+| TiKV      | i3.4xlarge|     3     |
+| TiDB      | c5.4xlarge|     3     |
+| TPC-C  | c5.9xlarge|     1     |
+
+### Software version
+
+| Service type   | Software version    |
+|:----------|:-----------|
+| PD        | 4.0 and 5.0   |
+| TiDB      | 4.0 and 5.0   |
+| TiKV      | 4.0 and 5.0   |
+| BenchmarkSQL  | None     |
+
+### Parameter configuration
+
+#### TiDB v4.0 configuration
+
+{{< copyable "" >}}
+
+```yaml
+log.level: "error"
+performance.max-procs: 20
+prepared-plan-cache.enabled: true
+tikv-client.max-batch-wait-time: 2000000
+```
+
+#### TiKV v4.0 configuration
+
+{{< copyable "" >}}
+
+```yaml
+pessimistic-txn.pipelined: true
+raftdb.allow-concurrent-memtable-write: true
+raftdb.max-background-jobs: 4
+raftstore.apply-max-batch-size: 2048
+raftstore.apply-pool-size: 3
+raftstore.store-max-batch-size: 2048
+raftstore.store-pool-size: 3
+readpool.storage.normal-concurrency: 10
+readpool.unified.max-thread-count: 20
+readpool.unified.min-thread-count: 5
+rocksdb.max-background-jobs: 8
+server.grpc-concurrency: 6
+server.request-batch-enable-cross-command: false
+storage.scheduler-worker-pool-size: 20
+```
+
+#### TiDB v5.0 configuration
+
+{{< copyable "" >}}
+
+```yaml
+log.level: "error"
+performance.max-procs: 20
+prepared-plan-cache.enabled: true
+tikv-client.max-batch-wait-time: 2000000
+```
+
+#### TiKV v5.0 configuration
+
+{{< copyable "" >}}
+
+```yaml
+pessimistic-txn.pipelined: true
+raftdb.allow-concurrent-memtable-write: true
+raftdb.max-background-jobs: 4
+raftstore.apply-max-batch-size: 2048
+raftstore.apply-pool-size: 3
+raftstore.store-max-batch-size: 2048
+raftstore.store-pool-size: 3
+readpool.storage.normal-concurrency: 10
+readpool.unified.max-thread-count: 20
+readpool.unified.min-thread-count: 5
+rocksdb.max-background-jobs: 8
+server.grpc-concurrency: 6
+server.request-batch-enable-cross-command: false
+storage.scheduler-worker-pool-size: 20
+```
+
+#### TiDB v4.0 global variable configuration
+
+{{< copyable "sql" >}}
+
+```sql
+set global tidb_hashagg_final_concurrency=1;
+set global tidb_hashagg_partial_concurrency=1;
+```
+
+#### TiDB v5.0 global variable configuration
+
+{{< copyable "sql" >}}
+
+```sql
+set global tidb_hashagg_final_concurrency=1;
+set global tidb_hashagg_partial_concurrency=1;
+set global tidb_enable_async_commit = 1;
+set global tidb_enable_1pc = 1;
+set global tidb_guarantee_linearizability = 0;
+set global tidb_enable_clustered_index = 1;
+```
+
+## Test plan
+
+1. Deploy TiDB v5.0 and v4.0 using TiUP.
+
+2. Use BenchmarkSQL to import the TPC-C 5000 Warehouse data.
+
+    1. Compile BenchmarkSQL：
+
+        {{< copyable "bash" >}}
+
+        ```bash
+        git clone https://github.com/pingcap/benchmarksql && cd benchmarksql && ant
+        ```
+
+    2. Enter the `run` directory, edit the `props.mysql` file according to the actual situation, and modify the `conn`, `warehouses`, `loadWorkers`, `terminals`, and `runMins` configuration items.
+
+    3. Execute the `runSQL.sh ./props.mysql sql.mysql/tableCreates.sql` command.
+
+    4. Execute the `runSQL.sh ./props.mysql sql.mysql/indexCreates.sql` command.
+
+    5. Run MySQL client and execute the `analyze table` statement on every table.
+
+3. Execute the `runBenchmark.sh ./props.mysql` command.
+
+4. Extract the tpmC data of New Order from the result.
+
+## Test result
+
+According to the test statistics, the TPC-C performance of TiDB v5.0 has **increased by 25%** compared with that of TiDB v4.0.
+
+![TPC-C](/media/tpcc_v5vsv4.png)
diff --git a/media/sysbench_v5vsv4_point_select.png b/media/sysbench_v5vsv4_point_select.png
diff --git a/media/sysbench_v5vsv4_read_write.png b/media/sysbench_v5vsv4_read_write.png
diff --git a/media/sysbench_v5vsv4_update_index.png b/media/sysbench_v5vsv4_update_index.png
diff --git a/media/sysbench_v5vsv4_update_non_index.png b/media/sysbench_v5vsv4_update_non_index.png
diff --git a/media/tpcc_v5vsv4.png b/media/tpcc_v5vsv4.png