Skip to content

Commit

Permalink
benchmark, faq, dev/reference: update the note for write conflict (pi…
Browse files Browse the repository at this point in the history 
…ngcap#1213)
  • Loading branch information
@TomShawn @jackysp
TomShawn authored and jackysp committed Jun 13, 2019
1 parent 7f264fe commit e1faa6f
Show file tree
Hide file tree
Showing 3 changed files with 55 additions and 49 deletions.
78 changes: 37 additions & 41 deletions benchmark/sysbench-v4.md → benchmark/how-to-run-sysbench.md
View file
Original file line number Diff line number Diff line change
@@ -1,33 +1,21 @@
---
title: Sysbench Test Result of TiDB 3.0 Beta on NVMe SSD
title: How to Test TiDB Using Sysbench
category: benchmark
---

# Sysbench Test Result of TiDB 3.0 Beta on NVMe SSD
# How to Test TiDB Using Sysbench

In this test, Sysbench 1.0.14 is used. It is recommended to use Sysbench 1.0 or later, which can be downloaded [here](https://github.com/akopytov/sysbench/releases/tag/1.0.14).

## Test purpose

This test aims to test the performance of TiDB 3.0 Beta on NVMe SSD.

## Test version, time, and place

TiDB version: 3.0 Beta

Time: February, 2019

Place: Beijing
In this test, Sysbench 1.0.14 and TiDB 3.0 Beta are used. It is recommended to use Sysbench 1.0 or later, which can be downloaded [here](https://github.com/akopytov/sysbench/releases/tag/1.0.14).

## Test environment

- [Hardware requirements](https://pingcap.com/docs/op-guide/recommendation/)

- The TiDB cluster is deployed according to the [TiDB Deployment Guide](https://pingcap.com/docs/op-guide/ansible-deployment/). Suppose there are 3 servers in total. It is recommended to deploy 1 TiDB instance, 1 PD instance and 1 TiKV instance on each server. As for disk space, supposing that there are 32 tables and 10M rows of data on each table, it is recommended that the disk space where TiKV's data directory resides is larger than 512 GB.
The number of concurrent connections to a single TiDB cluster is recommended to be under 500. If you need to increase the concurrency pressure on the entire system, you can add TiDB instances to the cluster whose number depends on the pressure of the test.
The number of concurrent connections to a single TiDB cluster is recommended to be under 500. If you need to increase the concurrency pressure on the entire system, you can add TiDB instances to the cluster whose number depends on the pressure of the test.

IDC machines:

| Type | Name |
|:---- |:---- |
| OS | Linux (CentOS 7.3.1611) |
Expand All @@ -38,7 +26,7 @@ IDC machines:

## Test plan

### TiDB version information
### TiDB version information

| Component | GitHash |
|:---- |:---- |
Expand All @@ -59,7 +47,7 @@ IDC machines:

Higher log level means fewer logs to be printed and thus positively influences TiDB performance. Enable `prepared plan cache` in the TiDB configuration to lower the cost of optimizing execution plan. Specifically, you can add the following command in the TiDB configuration file:

```
```toml
[log]
level = "error"
[prepared-plan-cache]
Expand All @@ -72,13 +60,13 @@ Higher log level also means better performance for TiKV.

As TiKV is deployed in clusters, the Raft algorithm can guarantee that data is written into most of the nodes. Therefore, except the scenarios where data security is extremely sensitive, `sync-log` can be disabled in raftstore.

There are 2 Column Families (Default CF and Write CF) on TiKV cluster which are mainly used to store different types of data. For the Sysbench test, the Column Family that is used to import data has a constant proportion among TiDB clusters:
There are 2 Column Families (Default CF and Write CF) on TiKV cluster which are mainly used to store different types of data. For the Sysbench test, the Column Family that is used to import data has a constant proportion among TiDB clusters:

Default CF : Write CF = 4 : 1

Configuring the block cache of RocksDB on TiKV should be based on the machine’s memory size, in order to make full use of the memory. To deploy a TiKV cluster on a 40GB virtual machine, it is suggested to configure the block cache as follows:

```
```toml
log-level = "error"
[raftstore]
sync-log = false
Expand All @@ -89,7 +77,7 @@ block-cache-size = "6GB"
```

For more detailed information on TiKV performance tuning, see [Tune TiKV Performance](https://pingcap.com/docs/op-guide/tune-tikv/).

## Test process

> **Note:**
Expand All @@ -100,7 +88,7 @@ For more detailed information on TiKV performance tuning, see [Tune TiKV Perform

This is an example of the Sysbench configuration file:

```
```txt
mysql-host={TIDB_HOST}
mysql-port=4000
mysql-user=root
Expand All @@ -115,7 +103,7 @@ The above parameters can be adjusted according to actual needs. Among them, `TID

See the following as a sample **config** file:

```
```txt
mysql-host=172.16.30.33
mysql-port=4000
mysql-user=root
Expand All @@ -128,66 +116,74 @@ db-driver=mysql

### Data import

Create the `sbtest` database using the following statement in the MySQL:
Before importing the data, it is necessary to make some settings to TiDB. Execute the following command in MySQL client:

```sql
set global tidb_disable_txn_auto_retry = off;
```

Then exit the client. TiDB uses an optimistic transaction model that rolls back transactions when a concurrency conflict is found. Setting `tidb_disable_txn_auto_retry` to `off` turns on the automatic retry mechanism after meeting a transaction conflict, which can prevent Sysbench from quitting because of the transaction conflict error.

Restart MySQL client and execute the following SQL statement to create a database `sbtest`:

```
create database sbtest;
```

Adjust the order in which Sysbench scripts create indexes. Sysbench imports data in the order of "Build Table -> Insert Data -> Create Index", which takes more time for TiDB to import data. Users can adjust the order to speed up the import of data. Suppose that you use Sysbench version https://github.com/akopytov/sysbench/tree/1.0.14. You can adjust the order in the following two ways.

1. Download the TiDB-modified [oltp_common.lua](https://raw.githubusercontent.com/pingcap/tidb-bench/master/sysbench/sysbench-patch/oltp_common.lua) file and overwrite the `/usr/share/sysbench/oltp_common.lua` file with it.
1. Download the TiDB-modified [oltp_common.lua](https://raw.githubusercontent.com/pingcap/tidb-bench/master/sysbench/sysbench-patch/oltp_common.lua) file and overwrite the `/usr/share/sysbench/oltp_common.lua` file with it.
2. Move the [235th](https://github.com/akopytov/sysbench/blob/1.0.14/src/lua/oltp_common.lua#L235) to [240th](https://github.com/akopytov/sysbench/blob/1.0.14/src/lua/oltp_common.lua#L240) lines of `/usr/share/sysbench/oltp_common.lua` to be right behind 198th lines.

> **Note:**
>
> This operation is optional and is only to save the time consumed by data import.
> This operation is optional and is only to save the time consumed by data import.
At the command line, enter the following command to start importing data. The config file is the one configured in the previous step:

```
```bash
sysbench --config-file=config oltp_point_select --tables=32 --table-size=10000000 prepare
```

### Warming data and collecting statistics

To warm data, we load data from disk into the block cache of memory. The warmed data has significantly improved the overall performance of the system. It is recommended to warm data once after restarting the cluster.

Sysbench does not provide data warming, so it must be done manually.
Sysbench does not provide data warming, so it must be done manually.

Take a table sbtest7 in Sysbench as an example. Execute the following SQL to warming up data:

```
```sql
SELECT COUNT(pad) FROM sbtest7 USE INDEX (k_7);
```

Collecting statistics helps the optimizer choose a more accurate execution plan. The `analyze` command can be used to collect statistics on the table sbtest. Each table needs statistics.

```
```sql
ANALYZE TABLE sbtest7;
```

### Point select test command

```
```bash
sysbench --config-file=config oltp_point_select --tables=32 --table-size=10000000 run
```

### Update index test command

```
```bash
sysbench --config-file=config oltp_update_index --tables=32 --table-size=10000000 run
```

### Read-only test command

```
```bash
sysbench --config-file=config oltp_read_only --tables=32 --table-size=10000000 run
```

## Test results

32 tables are tested, each with 10M of data.
32 tables are tested, each with 10M of data.

Sysbench test was carried on each of the tidb-servers. And the final result was a sum of all the results.

Expand All @@ -201,7 +197,7 @@ Sysbench test was carried on each of the tidb-servers. And the final result was
| point_select | 3\*64 | 195218.54 | 195218.54 | 0.98 | 2.14 | 21.82 |
| point_select | 3\*128 | 208189.53 | 208189.53 | 1.84 | 4.33 | 31.02 |

![](../media/oltp_point_select.png)
![oltp_point_select](/media/oltp_point_select.png)

### oltp_update_index

Expand All @@ -213,8 +209,8 @@ Sysbench test was carried on each of the tidb-servers. And the final result was
| oltp_update_index | 3\*64 | 18697.17 | 18697.17 | 10.34 | 17.63 | 4539.04 |
| oltp_update_index | 3\*128 | 20446.81 | 20446.81 | 18.98 | 40.37 | 5394.75 |
| oltp_update_index | 3\*256 | 23563.03 | 23563.03 | 32.86 | 78.60 | 5530.69 |
![](../media/oltp_update_index.png)

![oltp_update_index](/media/oltp_update_index.png)

### oltp_read_only

Expand All @@ -225,8 +221,8 @@ Sysbench test was carried on each of the tidb-servers. And the final result was
| oltp_read_only | 3\*32 | 5066.88 | 81070.16 | 19.42 | 26.20 | 123.41 |
| oltp_read_only | 3\*64 | 5466.36 | 87461.81 | 34.65 | 63.20 | 231.19 |
| oltp_read_only | 3\*128 | 6684.16 | 106946.59 | 57.29 | 97.55 | 180.85 |
![](../media/oltp_read_only.png)

![oltp_read_only](/media/oltp_read_only.png)

## Common issues

Expand All @@ -248,4 +244,4 @@ The actual CPU usage can be observed through Grafana's TiKV Thread CPU monitor p

CPU of NUMA architecture is used on some high-end equipment where cross-CPU access to remote memory will greatly reduce performance. By default, TiDB will use all CPUs of the server, and goroutine scheduling will inevitably lead to cross-CPU memory access.

Therefore, it is recommended to deploy *n* TiDBs (*n* is the number of NUMA CPUs) on the server of NUMA architecture, and meanwhile set the TiDB parameter `max-procs` to a value that is the same as the number of NUMA CPU cores.
Therefore, it is recommended to deploy *n* TiDBs (*n* is the number of NUMA CPUs) on the server of NUMA architecture, and meanwhile set the TiDB parameter `max-procs` to a value that is the same as the number of NUMA CPU cores.
4 changes: 3 additions & 1 deletion dev/reference/error-codes.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,15 @@ TiDB is compatible with the error codes in MySQL, and in most cases returns the
| 8001 | The memory used by the request exceeds the threshold limit for the TiDB memory usage. | Increase the value of the system variable with the `tidb_mem_quota` prefix. |
| 8002 | To guarantee consistency, a transaction with the `SELECT FOR UPDATE` statement cannot be retried when it encounters a commit conflict. TiDB rolls back the transaction and returns this error. | Retry the failed transaction. |
| 8003 | If the data in a row is not consistent with the index when executing the `ADMIN CHECK TABLE` command, TiDB returns this error. |
| 8004 | A single transaction is too large. | See [here](/faq/tidb.md#the-error-message-transaction-too-large-is-displayed) for the cause and solution. |
| 8005 | Transactions in TiDB encounter write conflicts. | See [here](/faq/tidb.md#troubleshoot) for the cause and solution. |
| 9001 | The PD request timed out. | Check the state/monitor/log of the PD server and the network between the TiDB server and the PD server. |
| 9002 | The TiKV request timed out. | Check the state/monitor/log of the TiKV server and the network between the TiDB server and the TiKV server. |
| 9003 | The TiKV server is busy and this usually occurs when the workload is too high. | Check the state/monitor/log of the TiKV server. |
| 9004 | This error occurs when a large number of transactional conflicts exist in the database. | Check the code of application. |
| 9005 | A certain Raft Group is not available, such as the number of replicas is not enough. This error usually occurs when the TiKV server is busy or the TiKV node is down. | Check the state/monitor/log of the TiKV server. |
| 9006 | The interval of GC Life Time is too short and the data that should be read by the long transactions might be cleared. | Extend the interval of GC Life Time. |
| 9500 | A single transaction is too large. | See [here](/faq/tidb.md#the-error-message-transaction-too-large-is-displayed) for the solution. |
| 9007 | Transactions in TiKV encounter write conflicts. | See [here](/faq/tidb.md#troubleshoot) for the cause and solution. |

## Troubleshooting

Expand Down
22 changes: 15 additions & 7 deletions faq/tidb.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -683,7 +683,7 @@ This is because the disk space is not enough. You need to add nodes or enlarge t
#### Why does the OOM (Out of Memory) error occur frequently in TiKV?
The memory usage of TiKV mainly comes from the block-cache of RocksDB, which is 40% of the system memory size by default. When the OOM error occurs frequently in TiKV, you should check whether the value of `block-cache-size` is set too high. In addition, when multiple TiKV instances are deployed on a single machine, you need to explicitly configure the parameter to prevent multiple instances from using too much system memory that results in the OOM error.
The memory usage of TiKV mainly comes from the block-cache of RocksDB, which is 40% of the system memory size by default. When the OOM error occurs frequently in TiKV, you should check whether the value of `block-cache-size` is set too high. In addition, when multiple TiKV instances are deployed on a single machine, you need to explicitly configure the parameter to prevent multiple instances from using too much system memory that results in the OOM error.
### TiDB test
Expand Down Expand Up @@ -725,7 +725,7 @@ See [mydumper Instructions](/tools/mydumper.md).
#### Loader
See [Loader Instructions](/tools/loader.md).
#### How to migrate an application running on MySQL to TiDB?
Because TiDB supports most MySQL syntax, generally you can migrate your applications to TiDB without changing a single line of code in most cases.
Expand Down Expand Up @@ -770,8 +770,8 @@ INSERT INTO mysql.user VALUES ("%", "root", "", "Y", "Y", "Y", "Y", "Y", "Y", "Y
```
#### Can TiDB provide services while Loader is running?
TiDB can provide services while Loader is running because Loader inserts the data logically. But do not perform the related DDL operations.
TiDB can provide services while Loader is running because Loader inserts the data logically. But do not perform the related DDL operations.
#### How to export the data in TiDB?
Expand Down Expand Up @@ -799,7 +799,7 @@ Two solutions:
- Add the `-Dsqoop.export.records.per.statement=10` option as follows:
```
```bash
sqoop export \
-Dsqoop.export.records.per.statement=10 \
--connect jdbc:mysql://mysql.example.com/sqoop \
Expand Down Expand Up @@ -1023,7 +1023,7 @@ The monitoring system of TiDB consists of Prometheus and Grafana. From the dashb
Yes. Find the startup script on the machine where Prometheus is started, edit the startup parameter and restart Prometheus.
```
```config
--storage.tsdb.retention="60d"
```
Expand All @@ -1045,6 +1045,10 @@ The `Statement OPS` statistics is only about applications related SQL statements
### TiDB custom error messages
#### ERROR 8005 (HY000): Write Conflict, txnStartTS is stale
Check whether `tidb_disable_txn_auto_retry` is set to `on`. If so, set it to `off`; if it is already `off`, increase the value of `tidb_retry_limit` until the error no longer occurs.
#### ERROR 9001 (HY000): PD Server Timeout
A PD request timeout. Check the status, monitoring data and log of the PD server, and the network between the TiDB server and the PD server.
Expand All @@ -1069,14 +1073,18 @@ The accessed Region is not available. A Raft Group is not available, with possib
The interval of `GC Life Time` is too short. The data that should have been read by long transactions might be deleted. You can add `GC Life Time` using the following command:
```
```sql
update mysql.tidb set variable_value='30m' where variable_name='tikv_gc_life_time';
```
> **Note:**
>
> "30m" means only cleaning up the data generated 30 minutes ago, which might consume some extra storage space.
#### ERROR 9007 (HY000): Write Conflict
Check whether `tidb_disable_txn_auto_retry` is set to `on`. If so, set it to `off`; if it is already `off`, increase the value of `tidb_retry_limit` until the error no longer occurs.
### MySQL native error messages
#### ERROR 2013 (HY000): Lost connection to MySQL server during query
Expand Down

0 comments on commit e1faa6f

Please sign in to comment.