benchmark, TOC: add 4.0 TPC-H, TPC-C and Sysbench test reports #2661
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
TomShawn
added
translation/from-docs-cn
TomShawn
changed the title
JaySon-Huang
reviewed
Jun 3, 2020
| @@ -1,113 +1,133 @@ | |||
| --- | |||
| title: TiDB TPC-C Performance Test Report -- v4.0 vs. v3.0 | |||
| summary: Compares the TPC-C performance of TiDB 4.0 and TiDB 3.0 using BenchmarkSQL. | |||
| summary: Compare the TPC-H performance of TiDB 4.0 and TiDB 3.0 using BenchmarkSQL. | |||
| git clone https://github.com/pingcap/tidb-bench.git && cd tpch/queries | ||
| ``` | ||
|
|
||
| 2. Execute the query and record the executing time of the query. |
There was a problem hiding this comment.
Should be "Execute those queries and record the executing time of each query"?
|
|
||
| + Blue lines represent v3.0; | ||
| + Red lines represent v4.0 (data read only from TiKV); | ||
| + Yellow lines represent v4.0 (data read from TiKV and TiFlash automatically based on intelligent choice). |
There was a problem hiding this comment.
We use the term "smart selection" instead of "intelligent choice" in this doc:
https://pingcap.com/docs/stable/tiflash/use-tiflash/#smart-selection
There was a problem hiding this comment.
This TiFlash overview document uses Intelligent choice https://pingcap.com/docs/stable/tiflash/tiflash-overview/#intelligent-choice
|
Please help review this PR, thx~ @CaitinChen FYI @TomShawn |
CaitinChen
changed the title
|
@JaySon-Huang, @scsldb, @CaitinChen, @shuke987, PTAL. |
CaitinChen
reviewed
Jun 5, 2020
There was a problem hiding this comment.
My review on the TPC-H doc is done. But I think we need to add numbers to show the improvements.
And, just like other benchmark reports, we should point put "TiDB version: v3.0.13 vs. v4.0.0."
@JaySon-Huang PTAL
| title: TiDB TPC-H Performance Test Report -- v4.0 vs. v3.0 | ||
| summary: Compare the TPC-H performance of TiDB 4.0 and TiDB 3.0. | ||
| category: benchmark | ||
| draft: true |
|
|
||
| ## Test purpose | ||
|
|
||
| This test aims to compare the TPC-H performances of TiDB 4.0 and TiDB 3.0 in the OLAP (Online Analytical Processing) scenario. |
There was a problem hiding this comment.
Suggested change
| This test aims to compare the TPC-H performances of TiDB 4.0 and TiDB 3.0 in the OLAP (Online Analytical Processing) scenario. | |
| This test aims to compare the TPC-H performances of TiDB 4.0 and TiDB 3.0 in the online analytical processing (OLAP) scenario. |
|
|
||
| This test aims to compare the TPC-H performances of TiDB 4.0 and TiDB 3.0 in the OLAP (Online Analytical Processing) scenario. | ||
|
|
||
| Because [TiFlash](/tiflash/tiflash-overview.md) is introduced in TiDB v4.0 that enhances the HTAP (Hybrid Transactional and Analytical Processing) performance of TiDB, the following performances are tested and compared in this report: |
There was a problem hiding this comment.
Suggested change
| Because [TiFlash](/tiflash/tiflash-overview.md) is introduced in TiDB v4.0 that enhances the HTAP (Hybrid Transactional and Analytical Processing) performance of TiDB, the following performances are tested and compared in this report: | |
| Because [TiFlash](/tiflash/tiflash-overview.md) is introduced in TiDB v4.0, which enhances TiDB's Hybrid Transactional and Analytical Processing (HTAP) capabilities, test objects in this report are as follows: |
There was a problem hiding this comment.
|
|
||
| #### Hardware prerequisite | ||
|
|
||
| To avoid TiKV and TiFlash racing for disk and I/O resources, mount the two NVMe SSD disks configured on EC2 as `/data1` and `/data2`. Deploy TiKV on `/data1` and deploy TiFlash on `/data2`. |
There was a problem hiding this comment.
Suggested change
| To avoid TiKV and TiFlash racing for disk and I/O resources, mount the two NVMe SSD disks configured on EC2 as `/data1` and `/data2`. Deploy TiKV on `/data1` and deploy TiFlash on `/data2`. | |
| To avoid TiKV and TiFlash racing for disk and I/O resources, mount the two NVMe SSD disks configured on EC2 to `/data1` and `/data2`. Deploy TiKV on `/data1` and deploy TiFlash on `/data2`. |
There was a problem hiding this comment.
Maybe "mount ... at "?
http://linuxcommand.org/lc3_man_pages/mount8.html
This tells the kernel to attach the filesystem found on device (which is of type type) at the directory dir.
|
|
||
| | Query ID | v3.0 | v4.0 TiKV Only | v4.0 TiKV/TiFlash Automatically | | ||
| | :-------- | :----------- | :------------ | :-------------- | | ||
| | 1 | 7.78s | 7.45s | 2.09s | |
There was a problem hiding this comment.
Suggested change
| | 1 | 7.78s | 7.45s | 2.09s | | |
| | 1 | 7.78 s | 7.45 s | 2.09 s | |
There was a problem hiding this comment.
Same comments for the following lines.
| | 21 | 11.12s | 6.23s | 6.26s | | ||
| | 22 | 4.49s | 3.05s | 2.31s | | ||
|
|
||
|  |
There was a problem hiding this comment.
Suggested change
|  | |
|  |
There was a problem hiding this comment.
"_" is not allowed in image names.
| + Blue lines represent v3.0; | ||
| + Red lines represent v4.0 (data read only from TiKV); | ||
| + Yellow lines represent v4.0 (data read from TiKV and TiFlash automatically based on intelligent choice). | ||
| + The ordinate represents the processing time of the query. The lower the ordinate, the better the performance. |
There was a problem hiding this comment.
Suggested change
| + The ordinate represents the processing time of the query. The lower the ordinate, the better the performance. | |
| + The y-axis represents the execution time of the query. The less the time, the better the performance. |
|
|
||
| Result description: | ||
|
|
||
| + **v4.0 TiKV Only** means that TiDB reads data only from TiKV. Comparing this result with that of v3.0, you can learn the TPC-H performance increase after TiDB and TiKV are upgraded to v4.0. |
There was a problem hiding this comment.
Suggested change
| + **v4.0 TiKV Only** means that TiDB reads data only from TiKV. Comparing this result with that of v3.0, you can learn the TPC-H performance increase after TiDB and TiKV are upgraded to v4.0. | |
| + **v4.0 TiKV Only** means that TiDB reads data only from TiKV. The result shows that the TPC-H performance increased after TiDB and TiKV are upgraded to v4.0. |
| Result description: | ||
|
|
||
| + **v4.0 TiKV Only** means that TiDB reads data only from TiKV. Comparing this result with that of v3.0, you can learn the TPC-H performance increase after TiDB and TiKV are upgraded to v4.0. | ||
| + **v4.0 TiKV/TiFlash Automatically** means that the TiDB optimizer automatically determines whether to use the TiFlash replica according to the cost estimation. Comparing this result with that of v3.0, you can learn the TPC-H performance increase in the full HTAP form of v4.0. |
There was a problem hiding this comment.
Suggested change
| + **v4.0 TiKV/TiFlash Automatically** means that the TiDB optimizer automatically determines whether to use the TiFlash replica according to the cost estimation. Comparing this result with that of v3.0, you can learn the TPC-H performance increase in the full HTAP form of v4.0. | |
| + **v4.0 TiKV/TiFlash Automatically** means that the TiDB optimizer automatically determines whether to read data from the TiFlash replica according to the cost estimation. The result shows that the TPC-H performance increased in the full HTAP form of v4.0. |
CaitinChen
reviewed
Jun 5, 2020
| --- | ||
|
|
||
| # TiDB TPC-C Performance Test Report -- v4.0 vs. v3.0 | ||
|
|
||
| ## Test purpose | ||
|
|
||
| This test aims to compare the TPC-C performance of TiDB 4.0 and TiDB 3.0. | ||
| This test aims to compare the TPC-C performances of TiDB 4.0 and TiDB 3.0 in the OLTP (Online Transactional Processing) scenario. |
There was a problem hiding this comment.
Suggested change
| This test aims to compare the TPC-C performances of TiDB 4.0 and TiDB 3.0 in the OLTP (Online Transactional Processing) scenario. | |
| This test aims to compare the TPC-C performances of TiDB 4.0 and TiDB 3.0 in the Online Transactional Processing (OLTP) scenario. |
| git clone https://github.com/pingcap/benchmarksql && cd benchmarksql && ant | ||
| ``` | ||
|
|
||
| 2. Enter the `run` directory, edit the `props.mysql` file according to the actual situation. Modify the `conn`, `warehouses`, `loadWorkers`, `terminals`, and `runMins` configuration items. |
There was a problem hiding this comment.
Suggested change
| 2. Enter the `run` directory, edit the `props.mysql` file according to the actual situation. Modify the `conn`, `warehouses`, `loadWorkers`, `terminals`, and `runMins` configuration items. | |
| 2. Enter the `run` directory, edit the `props.mysql` file according to the actual situation, and modify the `conn`, `warehouses`, `loadWorkers`, `terminals`, and `runMins` configuration items. |
|
|
||
| 4. Execute the `runSQL.sh ./props.mysql sql.mysql/indexCreates.sql` command. | ||
|
|
||
| 5. Execute MySQL client and execute the `analyze table` statement on every table. |
There was a problem hiding this comment.
Suggested change
| 5. Execute MySQL client and execute the `analyze table` statement on every table. | |
| 5. Run the MySQL client and execute the `analyze table` statement on every table. |
|
|
||
| According to the testing statistics, the performance of TiDB 4.0.0-rc **has increased by 45%** than that of TiDB 3.0.8. | ||
|  |
There was a problem hiding this comment.
Suggested change
|  | |
|  |
There was a problem hiding this comment.
"_" is not allowed in our figure names.
CaitinChen
reviewed
Jun 5, 2020
|
|
||
| ## Test purpose | ||
|
|
||
| This test aims to compare the Sysbench performances of TiDB 4.0 and TiDB 3.0 in the OLTP (Online Transactional Processing) scenario. |
There was a problem hiding this comment.
Suggested change
| This test aims to compare the Sysbench performances of TiDB 4.0 and TiDB 3.0 in the OLTP (Online Transactional Processing) scenario. | |
| This test aims to compare the Sysbench performances of TiDB 4.0 and TiDB 3.0 in the Online Transactional Processing (OLTP) scenario. |
|
|
||
| | Service type | Software version | | ||
| |:----------|:-----------| | ||
| | PD | 3.0、4.0 | |
There was a problem hiding this comment.
Suggested change
| | PD | 3.0、4.0 | | |
| | PD | 3.0 and 4.0 | |
| | Service type | Software version | | ||
| |:----------|:-----------| | ||
| | PD | 3.0、4.0 | | ||
| | TiDB | 3.0、4.0 | |
There was a problem hiding this comment.
Suggested change
| | TiDB | 3.0、4.0 | | |
| | TiDB | 3.0 and 4.0 | |
| |:----------|:-----------| | ||
| | PD | 3.0、4.0 | | ||
| | TiDB | 3.0、4.0 | | ||
| | TiKV | 3.0、4.0 | |
There was a problem hiding this comment.
Suggested change
| | TiKV | 3.0、4.0 | | |
| | TiKV | 3.0 and 4.0 | |
| {{< copyable "" >}} | ||
|
|
||
| ```yaml | ||
| log.level: “error” |
There was a problem hiding this comment.
Suggested change
| log.level: “error” | |
| log.level: "error" |
|
|
||
| Compared with v3.0, the Point Select performance of TiDB v4.0 has increased by 14%. | ||
|
|
||
|  |
There was a problem hiding this comment.
Suggested change
|  | |
|  |
There was a problem hiding this comment.
"_" is not allowed in our figure names.
|
|
||
| Compared with v3.0, the Update Non-index performance of TiDB v4.0 has increased by 15%. | ||
|
|
||
|  |
There was a problem hiding this comment.
Suggested change
|  | |
|  |
|
|
||
| Compared with v3.0, the Update Index performance of TiDB v4.0 has increased by 17%. | ||
|
|
||
|  |
There was a problem hiding this comment.
Suggested change
|  | |
|  |
|
|
||
|  | ||
|
|
||
| ### Read Write performance |
There was a problem hiding this comment.
Suggested change
| ### Read Write performance | |
| ### Read-write performance |
|
|
||
| ### Read Write performance | ||
|
|
||
| Compared with v3.0, the Read Write performance of TiDB v4.0 has increased by 31%. |
There was a problem hiding this comment.
Suggested change
| Compared with v3.0, the Read Write performance of TiDB v4.0 has increased by 31%. | |
| Compared with v3.0, the read-write performance of TiDB v4.0 has increased by 31%. |
|
@CaitinChen @JaySon-Huang Comments addressed, PTAL again. |
CaitinChen
reviewed
Jun 5, 2020
|
|
||
| ## Test result | ||
| ### Test result |
There was a problem hiding this comment.
Suggested change
| ### Test result | |
| ## Test result |
| ``` | ||
|
|
||
| ### Cluster topology | ||
| ### Test plan |
There was a problem hiding this comment.
Suggested change
| ### Test plan | |
| ## Test plan |
| set global tidb_index_lookup_join_concurrency = 16; | ||
| ``` | ||
|
|
||
| ### Test plan |
There was a problem hiding this comment.
Suggested change
| ### Test plan | |
| ## Test plan |
|
|
||
| ### Test plan | ||
|
|
||
| #### Hardware prerequisite |
There was a problem hiding this comment.
Suggested change
| #### Hardware prerequisite | |
| ### Hardware prerequisite |
|
|
||
| To avoid TiKV and TiFlash racing for disk and I/O resources, mount the two NVMe SSD disks configured on EC2 to `/data1` and `/data2`. Deploy TiKV on `/data1` and deploy TiFlash on `/data2`. | ||
|
|
||
| #### Test process |
There was a problem hiding this comment.
Suggested change
| #### Test process | |
| ### Test process |
|
|
||
| 4. Extract and organize the data of query execution time. | ||
|
|
||
| ### Test result |
There was a problem hiding this comment.
Suggested change
| ### Test result | |
| ## Test result |
sre-bot
pushed a commit
to sre-bot/docs
that referenced
this pull request
Jun 5, 2020
Signed-off-by: sre-bot <sre-bot@pingcap.com>
5 tasks
|
cherry pick to release-4.0 in PR #2711 |
TomShawn
added a commit
that referenced
this pull request
Jun 5, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is changed, added or deleted? (Required)
add 4.0 TPC-H， TPC-C and Sysbench test reports.
Which TiDB version(s) do your changes apply to? (Required)
If you select two or more versions from above, to trigger the bot to cherry-pick this PR to your desired release version branch(es), you must add corresponding labels such as needs-cherry-pick-4.0, needs-cherry-pick-3.1, needs-cherry-pick-3.0, and needs-cherry-pick-2.1.
What is the related PR or file link(s)?