Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

how-to/maintain: fix the description of backup & restore document #1416

Merged
merged 17 commits into from
Aug 6, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 41 additions & 13 deletions dev/how-to/maintain/backup-and-restore.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,6 @@ category: how-to

# Backup and Restore

## About

This document describes how to back up and restore the data of TiDB. Currently, this document only covers full backup and restoration.

Here we assume that the TiDB service information is as follows:
Expand All @@ -21,7 +19,7 @@ Use the following tools for data backup and restoration:
- `mydumper`: to export data from TiDB
- `loader`: to import data into TiDB

### Download TiDB toolset (Linux)
## Download TiDB toolset (Linux)

```bash
# Download the tool package.
Expand All @@ -38,7 +36,7 @@ cd tidb-enterprise-tools-latest-linux-amd64

## Full backup and restoration using `mydumper`/`loader`

You can use [`mydumper`](/reference/tools/mydumper.md) to export data from MySQL and [`loader`](/reference/tools/loader.md) to import the data into TiDB.
You can use [`mydumper`](/reference/tools/mydumper.md) to export data from TiDB and [`loader`](/reference/tools/loader.md) to import the data into TiDB.

> **Important**: You must use the `mydumper` from the Enterprise Tools package, and not the `mydumper` provided by your operating system's package manager. The upstream version of `mydumper` does not yet handle TiDB correctly ([#155](https://github.com/maxbube/mydumper/pull/155)). Using `mysqldump` is also not recommended, as it is much slower for both backup and restoration.

Expand All @@ -47,21 +45,21 @@ You can use [`mydumper`](/reference/tools/mydumper.md) to export data from MySQL
To quickly backup and restore data (especially large amounts of data), refer to the following recommendations:

- Keep the exported data file as small as possible and it is recommended keep it within 64M. You can use the `-F` parameter to set the value.
- You can adjust the `-t` parameter of `loader` based on the number and the load of TiKV instances. For example, if there are three TiKV instances, `-t` can be set to 3 * (1 ~ n). If the load of TiKV is too high and the log `backoffer.maxSleep 15000ms is exceeded` is displayed many times, decrease the value of `-t`; otherwise, increase it.
- You can adjust the `-t` parameter of `loader` based on the number and the load of TiKV instances. For example, if there are three TiKV instances, `-t` can be set to around 3 * (1 ~ n). If the load of TiKV is too high and the log `backoffer.maxSleep 15000ms is exceeded` is displayed many times, decrease the value of `-t`; otherwise, increase it.

#### An example of restoring data and related configuration

- The total size of the exported files is 214G. A single table has 8 columns and 2 billion rows.
- The cluster topology:
- 12 TiKV instances: 4 nodes, 3 TiKV instances per node
- 4 TiDB instances
- 3 PD instances
- 12 TiKV instances: 4 nodes, 3 TiKV instances per node
- 4 TiDB instances
- 3 PD instances
- The configuration of each node:
- CPU: Intel Xeon E5-2670 v3 @ 2.30GHz
- 48 vCPU [2 x 12 physical cores]
- Memory: 128G
- Disk: sda [raid 10, 300G] sdb[RAID 5, 2T]
- Operating System: CentOS 7.3
- CPU: Intel Xeon E5-2670 v3 @ 2.30GHz
- 48 vCPU [2 x 12 physical cores]
- Memory: 128G
- Disk: sda [raid 10, 300G] sdb[RAID 5, 2T]
- Operating System: CentOS 7.3
- The `-F` parameter of `mydumper` is set to 16 and the `-t` parameter of `loader` is set to 64.

**Results**: It takes 11 hours to import all the data, which is 19.4G/hour.
Expand All @@ -82,6 +80,36 @@ In this command,
- `-F 64`: means a table is partitioned into chunks and one chunk is 64MB.
- `--skip-tz-utc`: the purpose of adding this parameter is to ignore the inconsistency of time zone setting between MySQL and the data exporting machine and to disable automatic conversion.

If `mydumper` emits error like:

```
** (mydumper:27528): CRITICAL **: 13:25:09.081: Could not read data from testSchema.testTable: GC life time is shorter than transaction duration, transaction starts at 2019-08-05 21:10:01.451 +0800 CST, GC safe point is 2019-08-05 21:14:53.801 +0800 CST
```

Then execute two more commands:

- Step 1: before executing the `mydumper` command, query the GC values of the TiDB cluster and adjust it to a suitable value using the MySQL client.

```sql
mysql> SELECT * FROM mysql.tidb WHERE VARIABLE_NAME = 'tikv_gc_life_time'
+-----------------------+------------------------------------------------------------------------------------------------+
| VARIABLE_NAME | VARIABLE_VALUE |
+-----------------------+------------------------------------------------------------------------------------------------+
| tikv_gc_life_time | 10m0s |
+-----------------------+------------------------------------------------------------------------------------------------+
1 rows in set (0.02 sec)

mysql> update mysql.tidb set VARIABLE_VALUE = '720h' where VARIABLE_NAME = 'tikv_gc_life_time';
```

- Step 2: after you finish running the `mydumper` command, restore the GC value of the TiDB cluster to its original value in step 1.

{{< copyable "sql" >}}

```sql
update mysql.tidb set VARIABLE_VALUE = '10m' where VARIABLE_NAME = 'tikv_gc_life_time';
```

### Restore data into TiDB

To restore data into TiDB, use `loader` to import the previously exported data. See [Loader instructions](/reference/tools/loader.md) for more information.
Expand Down
54 changes: 41 additions & 13 deletions v2.1/how-to/maintain/backup-and-restore.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,6 @@ category: how-to

# Backup and Restore

## About

This document describes how to back up and restore the data of TiDB. Currently, this document only covers full backup and restoration.

Here we assume that the TiDB service information is as follows:
Expand All @@ -21,7 +19,7 @@ Use the following tools for data backup and restoration:
- `mydumper`: to export data from TiDB
- `loader`: to import data into TiDB

### Download TiDB toolset (Linux)
## Download TiDB toolset (Linux)

```bash
# Download the tool package.
Expand All @@ -38,7 +36,7 @@ cd tidb-enterprise-tools-latest-linux-amd64

## Full backup and restoration using `mydumper`/`loader`

You can use [`mydumper`](/reference/tools/mydumper.md) to export data from MySQL and [`loader`](/reference/tools/loader.md) to import the data into TiDB.
You can use [`mydumper`](/reference/tools/mydumper.md) to export data from TiDB and [`loader`](/reference/tools/loader.md) to import the data into TiDB.

> **Important**: You must use the `mydumper` from the Enterprise Tools package, and not the `mydumper` provided by your operating system's package manager. The upstream version of `mydumper` does not yet handle TiDB correctly ([#155](https://github.com/maxbube/mydumper/pull/155)). Using `mysqldump` is also not recommended, as it is much slower for both backup and restoration.

Expand All @@ -47,21 +45,21 @@ You can use [`mydumper`](/reference/tools/mydumper.md) to export data from MySQL
To quickly backup and restore data (especially large amounts of data), refer to the following recommendations:

- Keep the exported data file as small as possible and it is recommended keep it within 64M. You can use the `-F` parameter to set the value.
- You can adjust the `-t` parameter of `loader` based on the number and the load of TiKV instances. For example, if there are three TiKV instances, `-t` can be set to 3 * (1 ~ n). If the load of TiKV is too high and the log `backoffer.maxSleep 15000ms is exceeded` is displayed many times, decrease the value of `-t`; otherwise, increase it.
- You can adjust the `-t` parameter of `loader` based on the number and the load of TiKV instances. For example, if there are three TiKV instances, `-t` can be set to around 3 * (1 ~ n). If the load of TiKV is too high and the log `backoffer.maxSleep 15000ms is exceeded` is displayed many times, decrease the value of `-t`; otherwise, increase it.

#### An example of restoring data and related configuration

- The total size of the exported files is 214G. A single table has 8 columns and 2 billion rows.
- The cluster topology:
- 12 TiKV instances: 4 nodes, 3 TiKV instances per node
- 4 TiDB instances
- 3 PD instances
- 12 TiKV instances: 4 nodes, 3 TiKV instances per node
- 4 TiDB instances
- 3 PD instances
- The configuration of each node:
- CPU: Intel Xeon E5-2670 v3 @ 2.30GHz
- 48 vCPU [2 x 12 physical cores]
- Memory: 128G
- Disk: sda [raid 10, 300G] sdb[RAID 5, 2T]
- Operating System: CentOS 7.3
- CPU: Intel Xeon E5-2670 v3 @ 2.30GHz
- 48 vCPU [2 x 12 physical cores]
- Memory: 128G
- Disk: sda [raid 10, 300G] sdb[RAID 5, 2T]
- Operating System: CentOS 7.3
- The `-F` parameter of `mydumper` is set to 16 and the `-t` parameter of `loader` is set to 64.

**Results**: It takes 11 hours to import all the data, which is 19.4G/hour.
Expand All @@ -82,6 +80,36 @@ In this command,
- `-F 64`: means a table is partitioned into chunks and one chunk is 64MB.
- `--skip-tz-utc`: the purpose of adding this parameter is to ignore the inconsistency of time zone setting between MySQL and the data exporting machine and to disable automatic conversion.

If `mydumper` emits error like:

```
** (mydumper:27528): CRITICAL **: 13:25:09.081: Could not read data from testSchema.testTable: GC life time is shorter than transaction duration, transaction starts at 2019-08-05 21:10:01.451 +0800 CST, GC safe point is 2019-08-05 21:14:53.801 +0800 CST
```

Then execute two more commands:

- Step 1: before executing the `mydumper` command, query the GC values of the TiDB cluster and adjust it to a suitable value using the MySQL client.

```sql
mysql> SELECT * FROM mysql.tidb WHERE VARIABLE_NAME = 'tikv_gc_life_time'
+-----------------------+------------------------------------------------------------------------------------------------+
| VARIABLE_NAME | VARIABLE_VALUE |
+-----------------------+------------------------------------------------------------------------------------------------+
| tikv_gc_life_time | 10m0s |
+-----------------------+------------------------------------------------------------------------------------------------+
1 rows in set (0.02 sec)

mysql> update mysql.tidb set VARIABLE_VALUE = '720h' where VARIABLE_NAME = 'tikv_gc_life_time';
```

- Step 2: after you finish running the `mydumper` command, restore the GC value of the TiDB cluster to its original value in step 1.

{{< copyable "sql" >}}

```sql
update mysql.tidb set VARIABLE_VALUE = '10m' where VARIABLE_NAME = 'tikv_gc_life_time';
```

### Restore data into TiDB

To restore data into TiDB, use `loader` to import the previously exported data. See [Loader instructions](/reference/tools/loader.md) for more information.
Expand Down
54 changes: 41 additions & 13 deletions v3.0/how-to/maintain/backup-and-restore.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,6 @@ aliases: ['/docs/op-guide/backup-restore/']

# Backup and Restore

## About

This document describes how to back up and restore the data of TiDB. Currently, this document only covers full backup and restoration.

Here we assume that the TiDB service information is as follows:
Expand All @@ -22,7 +20,7 @@ Use the following tools for data backup and restoration:
- `mydumper`: to export data from TiDB
- `loader`: to import data into TiDB

### Download TiDB toolset (Linux)
## Download TiDB toolset (Linux)

```bash
# Download the tool package.
Expand All @@ -39,7 +37,7 @@ cd tidb-enterprise-tools-latest-linux-amd64

## Full backup and restoration using `mydumper`/`loader`

You can use [`mydumper`](/reference/tools/mydumper.md) to export data from MySQL and [`loader`](/reference/tools/loader.md) to import the data into TiDB.
You can use [`mydumper`](/reference/tools/mydumper.md) to export data from TiDB and [`loader`](/reference/tools/loader.md) to import the data into TiDB.

> **Important**: You must use the `mydumper` from the Enterprise Tools package, and not the `mydumper` provided by your operating system's package manager. The upstream version of `mydumper` does not yet handle TiDB correctly ([#155](https://github.com/maxbube/mydumper/pull/155)). Using `mysqldump` is also not recommended, as it is much slower for both backup and restoration.

Expand All @@ -48,21 +46,21 @@ You can use [`mydumper`](/reference/tools/mydumper.md) to export data from MySQL
To quickly backup and restore data (especially large amounts of data), refer to the following recommendations:

- Keep the exported data file as small as possible and it is recommended keep it within 64M. You can use the `-F` parameter to set the value.
- You can adjust the `-t` parameter of `loader` based on the number and the load of TiKV instances. For example, if there are three TiKV instances, `-t` can be set to 3 * (1 ~ n). If the load of TiKV is too high and the log `backoffer.maxSleep 15000ms is exceeded` is displayed many times, decrease the value of `-t`; otherwise, increase it.
- You can adjust the `-t` parameter of `loader` based on the number and the load of TiKV instances. For example, if there are three TiKV instances, `-t` can be set to around 3 * (1 ~ n). If the load of TiKV is too high and the log `backoffer.maxSleep 15000ms is exceeded` is displayed many times, decrease the value of `-t`; otherwise, increase it.

#### An example of restoring data and related configuration

- The total size of the exported files is 214G. A single table has 8 columns and 2 billion rows.
- The cluster topology:
- 12 TiKV instances: 4 nodes, 3 TiKV instances per node
- 4 TiDB instances
- 3 PD instances
- 12 TiKV instances: 4 nodes, 3 TiKV instances per node
- 4 TiDB instances
- 3 PD instances
- The configuration of each node:
- CPU: Intel Xeon E5-2670 v3 @ 2.30GHz
- 48 vCPU [2 x 12 physical cores]
- Memory: 128G
- Disk: sda [raid 10, 300G] sdb[RAID 5, 2T]
- Operating System: CentOS 7.3
- CPU: Intel Xeon E5-2670 v3 @ 2.30GHz
- 48 vCPU [2 x 12 physical cores]
- Memory: 128G
- Disk: sda [raid 10, 300G] sdb[RAID 5, 2T]
- Operating System: CentOS 7.3
- The `-F` parameter of `mydumper` is set to 16 and the `-t` parameter of `loader` is set to 64.

**Results**: It takes 11 hours to import all the data, which is 19.4G/hour.
Expand All @@ -83,6 +81,36 @@ In this command,
- `-F 64`: means a table is partitioned into chunks and one chunk is 64MB.
- `--skip-tz-utc`: the purpose of adding this parameter is to ignore the inconsistency of time zone setting between MySQL and the data exporting machine and to disable automatic conversion.

If `mydumper` emits error like:

```
** (mydumper:27528): CRITICAL **: 13:25:09.081: Could not read data from testSchema.testTable: GC life time is shorter than transaction duration, transaction starts at 2019-08-05 21:10:01.451 +0800 CST, GC safe point is 2019-08-05 21:14:53.801 +0800 CST
```

Then execute two more commands:

- Step 1: before executing the `mydumper` command, query the GC values of the TiDB cluster and adjust it to a suitable value using the MySQL client.

```sql
mysql> SELECT * FROM mysql.tidb WHERE VARIABLE_NAME = 'tikv_gc_life_time'
+-----------------------+------------------------------------------------------------------------------------------------+
| VARIABLE_NAME | VARIABLE_VALUE |
+-----------------------+------------------------------------------------------------------------------------------------+
| tikv_gc_life_time | 10m0s |
+-----------------------+------------------------------------------------------------------------------------------------+
1 rows in set (0.02 sec)

mysql> update mysql.tidb set VARIABLE_VALUE = '720h' where VARIABLE_NAME = 'tikv_gc_life_time';
```

- Step 2: after you finish running the `mydumper` command, restore the GC value of the TiDB cluster to its original value in step 1.

{{< copyable "sql" >}}

```sql
update mysql.tidb set VARIABLE_VALUE = '10m' where VARIABLE_NAME = 'tikv_gc_life_time';
```

### Restore data into TiDB

To restore data into TiDB, use `loader` to import the previously exported data. See [Loader instructions](/reference/tools/loader.md) for more information.
Expand Down