forked from pingcap/docs
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge docs special week branch to docs master (pingcap#2623)
* Update command line flags for pd configruation (pingcap#2596) Signed-off-by: nolouch <nolouch@gmail.com> * Remove upgrade-tiflash.md (pingcap#2597) * Remove upgrade-tiflash.md Duplicated with TiDB's upgrade manual. * delete a link Co-authored-by: yikeke <yikeke@pingcap.com> * Move TiFlash troubleshoot part to a new document (pingcap#2602) * best-practice: update massive regions to match 4.0 (pingcap#2595) Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * sql-statements: document BACKUP, RESTORE and SHOW [BACKUPS|RESTORES] (pingcap#2618) Co-authored-by: ShuNing <nolouch@gmail.com> Co-authored-by: Flowyi <flowbehappy@gmail.com> Co-authored-by: lei yu <leiysky@outlook.com> Co-authored-by: Jay <BusyJay@users.noreply.github.com> Co-authored-by: kennytm <kennytm@gmail.com>
- Loading branch information
6 people
authored
May 28, 2020
1 parent
180f7b0
commit 70df9ab
Showing
20 changed files
with
613 additions
and
124 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| --- | ||
| title: BR storages | ||
| summary: Describes the storage URL format used in BR. | ||
| category: reference | ||
| --- | ||
|
|
||
| # BR storages | ||
|
|
||
| BR supports reading and writing data on the local filesystem, as well as on Amazon S3 and Google Cloud Storage. These are distinguished by the URL scheme in the `--storage` parameter passed into BR. | ||
|
|
||
| ## Schemes | ||
|
|
||
| The following services are supported: | ||
|
|
||
| | Service | Schemes | Example URL | | ||
| |---------|---------|-------------| | ||
| | Local filesystem, distributed on every node | local | `local:///path/to/dest/` | | ||
| | Amazon S3 and compatible services | s3 | `s3://bucket-name/prefix/of/dest/` | | ||
| | Google Cloud Storage (GCS) | gcs, gs | `gcs://bucket-name/prefix/of/dest/` | | ||
| | Write to nowhere (for benchmarking only) | noop | `noop://` | | ||
|
|
||
| ## Parameters | ||
|
|
||
| Cloud storages such as S3 and GCS sometimes require additional configuration for connection. You can specify parameters for such configuration. For example: | ||
|
|
||
| {{< copyable "shell-regular" >}} | ||
|
|
||
| ```shell | ||
| ./br backup full -u 127.0.0.1:2379 -s 's3://bucket-name/prefix?region=us-west-2' | ||
| ``` | ||
|
|
||
| ### S3 parameters | ||
|
|
||
| | Parameter | Description | | ||
| |----------:|---------| | ||
| | `access-key` | The access key | | ||
| | `secret-access-key` | The secret access key | | ||
| | `region` | Service Region for Amazon S3 (default to `us-east-1`) | | ||
| | `use-accelerate-endpoint` | Whether to use the accelerate endpoint on Amazon S3 (default to `false`) | | ||
| | `endpoint` | URL of custom endpoint for S3-compatible services (for example, `https://s3.example.com/`) | | ||
| | `force-path-style` | Use path style access rather than virtual hosted style access (default to `false`) | | ||
| | `storage-class` | Storage class of the uploaded objects (for example, `STANDARD`, `STANDARD_IA`) | | ||
| | `sse` | Server-side encryption algorithm used to encrypt the upload (empty, `AES256` or `aws:kms`) | | ||
| | `sse-kms-key-id` | If `sse` is set to `aws:kms`, specifies the KMS ID | | ||
| | `acl` | Canned ACL of the uploaded objects (for example, `private`, `authenticated-read`) | | ||
|
|
||
| > **Note:** | ||
| > | ||
| > It is not recommended to pass in the access key and secret access key directly in the storage URL, because these keys are logged in plain text. BR tries to infer these keys from the environment in the following order: | ||
| 1. `$AWS_ACCESS_KEY_ID` and `$AWS_SECRET_ACCESS_KEY` environment variables | ||
| 2. `$AWS_ACCESS_KEY` and `$AWS_SECRET_KEY` environment variables | ||
| 3. Shared credentials file on the BR node at the path specified by the `$AWS_SHARED_CREDENTIALS_FILE` environment variable | ||
| 4. Shared credentials file on the BR node at `~/.aws/credentials` | ||
| 5. Current IAM role of the Amazon EC2 container | ||
| 6. Current IAM role of the Amazon ECS task | ||
|
|
||
| ### GCS parameters | ||
|
|
||
| | Parameter | Description | | ||
| |----------:|---------| | ||
| | `credentials-file` | The path to the credentials JSON file on the TiDB node | | ||
| | `storage-class` | Storage class of the uploaded objects (for example, `STANDARD`, `COLDLINE`) | | ||
| | `predefined-acl` | Predefined ACL of the uploaded objects (for example, `private`, `project-private`) | | ||
|
|
||
| When `credentials-file` is not specified, BR will try to infer the credentials from the environment, in the following order: | ||
|
|
||
| 1. Content of the file on the BR node at the path specified by the `$GOOGLE_APPLICATION_CREDENTIALS` environment variable | ||
| 2. Content of the file on the BR node at `~/.config/gcloud/application_default_credentials.json` | ||
| 3. When running in GCE or GAE, the credentials fetched from the metadata server. | ||
|
|
||
| ## Sending credentials to TiKV | ||
|
|
||
| By default, when using S3 and GCS destinations, BR will send the credentials to every TiKV nodes to reduce setup complexity. | ||
|
|
||
| However, this is unsuitable on cloud environment, where every node has their own role and permission. In such cases, you need to disable credentials sending with `--send-credentials-to-tikv=false` (or the short form `-c=0`): | ||
|
|
||
| {{< copyable "shell-regular" >}} | ||
|
|
||
| ```shell | ||
| ./br backup full -c=0 -u pd-service:2379 -s 's3://bucket-name/prefix' | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,190 @@ | ||
| --- | ||
| title: BACKUP | TiDB SQL Statement Reference | ||
| summary: An overview of the usage of BACKUP for the TiDB database. | ||
| category: reference | ||
| --- | ||
|
|
||
| # BACKUP | ||
|
|
||
| This statement is used to perform a distributed backup of the TiDB cluster. | ||
|
|
||
| The `BACKUP` statement uses the same engine as the [BR tool](/br/backup-and-restore-use-cases.md) does, except that the backup process is driven by TiDB itself rather than a separate BR tool. All benefits and warnings of BR also apply in this statement. | ||
|
|
||
| Executing `BACKUP` requires `SUPER` privilege. Additionally, both the TiDB node executing the backup and all TiKV nodes in the cluster must have read or write permission to the destination. | ||
|
|
||
| The `BACKUP` statement is blocked until the entire backup task is finished, failed, or canceled. A long-lasting connection should be prepared for executing `BACKUP`. The task can be canceled using the [`KILL TIDB QUERY`](/sql-statements/sql-statement-kill.md) statement. | ||
|
|
||
| Only one `BACKUP` and [`RESTORE`](/sql-statements/sql-statement-restore.md) task can be executed at a time. If a `BACKUP` or `RESTORE` statement is already being executed on the same TiDB server, the new `BACKUP` execution will wait until all previous tasks are finished. | ||
|
|
||
| `BACKUP` can only be used with "tikv" storage engine. Using `BACKUP` with the "mocktikv" engine will fail. | ||
|
|
||
| ## Synopsis | ||
|
|
||
| **BackupStmt:** | ||
|
|
||
|  | ||
|
|
||
| **BRIETables:** | ||
|
|
||
|  | ||
|
|
||
| **BackupOption:** | ||
|
|
||
|  | ||
|
|
||
| **Boolean:** | ||
|
|
||
|  | ||
|
|
||
| **BackupTSO:** | ||
|
|
||
|  | ||
|
|
||
| ## Examples | ||
|
|
||
| ### Back up databases | ||
|
|
||
| {{< copyable "sql" >}} | ||
|
|
||
| ```sql | ||
| BACKUP DATABASE `test` TO 'local:///mnt/backup/2020/04/'; | ||
| ``` | ||
|
|
||
| ```sql | ||
| +------------------------------+-----------+-----------------+---------------------+---------------------+ | ||
| | Destination | Size | BackupTS | Queue Time | Execution Time | | ||
| +------------------------------+-----------+-----------------+---------------------+---------------------+ | ||
| | local:///mnt/backup/2020/04/ | 248665063 | 416099531454472 | 2020-04-12 23:09:48 | 2020-04-12 23:09:48 | | ||
| +------------------------------+-----------+-----------------+---------------------+---------------------+ | ||
| 1 row in set (58.453 sec) | ||
| ``` | ||
|
|
||
| In the example above, the `test` database is backed up into the local filesystem. The data is saved as SST files in the `/mnt/backup/2020/04/` directories distributed among all TiDB and TiKV nodes. | ||
|
|
||
| The first row of the result above is described as follows: | ||
|
|
||
| | Column | Description | | ||
| | :-------- | :--------- | | ||
| | `Destination` | The destination URL | | ||
| | `Size` | The total size of the backup archive, in bytes | | ||
| | `BackupTS` | The TSO of the snapshot when the backup is created (useful for [incremental backup](#incremental-backup)) | | ||
| | `Queue Time` | The timestamp (in current time zone) when the `BACKUP` task is queued. | | ||
| | `Execution Time` | The timestamp (in current time zone) when the `BACKUP` task starts to run. | | ||
|
|
||
| ### Back up tables | ||
|
|
||
| {{< copyable "sql" >}} | ||
|
|
||
| ```sql | ||
| BACKUP TABLE `test`.`sbtest01` TO 'local:///mnt/backup/sbtest01/'; | ||
| ``` | ||
|
|
||
| {{< copyable "sql" >}} | ||
|
|
||
| ```sql | ||
| BACKUP TABLE sbtest02, sbtest03, sbtest04 TO 'local:///mnt/backup/sbtest/'; | ||
| ``` | ||
|
|
||
| ### Back up the entire cluster | ||
|
|
||
| {{< copyable "sql" >}} | ||
|
|
||
| ```sql | ||
| BACKUP DATABASE * TO 'local:///mnt/backup/full/'; | ||
| ``` | ||
|
|
||
| Note that the system tables (`mysql.*`, `INFORMATION_SCHEMA.*`, `PERFORMANCE_SCHEMA.*`, …) will not be included into the backup. | ||
|
|
||
| ### Remote destinations | ||
|
|
||
| BR supports backing up data to S3 or GCS: | ||
|
|
||
| {{< copyable "sql" >}} | ||
|
|
||
| ```sql | ||
| BACKUP DATABASE `test` TO 's3://example-bucket-2020/backup-05/?region=us-west-2'; | ||
| ``` | ||
|
|
||
| The URL syntax is further explained in [BR storages](/br/backup-and-restore-storages.md). | ||
|
|
||
| When running on cloud environment where credentials should not be distributed, set the `SEND_CREDENTIALS_TO_TIKV` option to `FALSE`: | ||
|
|
||
| {{< copyable "sql" >}} | ||
|
|
||
| ```sql | ||
| BACKUP DATABASE `test` TO 's3://example-bucket-2020/backup-05/?region=us-west-2' | ||
| SEND_CREDENTIALS_TO_TIKV = FALSE; | ||
| ``` | ||
|
|
||
| ### Performance fine-tuning | ||
|
|
||
| Use `RATE_LIMIT` to limit the average upload speed per TiKV node to reduce network bandwidth. | ||
|
|
||
| By default, every TiKV node would run 4 backup threads. This value can be adjusted with the `CONCURRENCY` option. | ||
|
|
||
| Before backup is completed, `BACKUP` would perform a checksum against the data on the cluster to verify correctness. This step can be disabled with the `CHECKSUM` option if you are confident that this is unnecessary. | ||
|
|
||
| {{< copyable "sql" >}} | ||
|
|
||
| ```sql | ||
| BACKUP DATABASE `test` TO 's3://example-bucket-2020/backup-06/' | ||
| RATE_LIMIT = 120 MB/SECOND | ||
| CONCURRENCY = 8 | ||
| CHECKSUM = FALSE; | ||
| ``` | ||
|
|
||
| ### Snapshot | ||
|
|
||
| Specify a timestamp, TSO or relative time to backup historical data. | ||
|
|
||
| {{< copyable "sql" >}} | ||
|
|
||
| ```sql | ||
| -- relative time | ||
| BACKUP DATABASE `test` TO 'local:///mnt/backup/hist01' | ||
| SNAPSHOT = 36 HOUR AGO; | ||
|
|
||
| -- timestamp (in current time zone) | ||
| BACKUP DATABASE `test` TO 'local:///mnt/backup/hist02' | ||
| SNAPSHOT = '2020-04-01 12:00:00'; | ||
|
|
||
| -- timestamp oracle | ||
| BACKUP DATABASE `test` TO 'local:///mnt/backup/hist03' | ||
| SNAPSHOT = 415685305958400; | ||
| ``` | ||
|
|
||
| The supported units for relative time are: | ||
|
|
||
| * MICROSECOND | ||
| * SECOND | ||
| * MINUTE | ||
| * HOUR | ||
| * DAY | ||
| * WEEK | ||
|
|
||
| Note that, following SQL standard, the units are always singular. | ||
|
|
||
| ### Incremental backup | ||
|
|
||
| Supply the `LAST_BACKUP` option to only backup the changes between the last backup to the current snapshot. | ||
|
|
||
| {{< copyable "sql" >}} | ||
|
|
||
| ```sql | ||
| -- timestamp (in current time zone) | ||
| BACKUP DATABASE `test` TO 'local:///mnt/backup/hist02' | ||
| LAST_BACKUP = '2020-04-01 12:00:00'; | ||
|
|
||
| -- timestamp oracle | ||
| BACKUP DATABASE `test` TO 'local:///mnt/backup/hist03' | ||
| LAST_BACKUP = 415685305958400; | ||
| ``` | ||
|
|
||
| ## MySQL compatibility | ||
|
|
||
| This statement is a TiDB extension to MySQL syntax. | ||
|
|
||
| ## See also | ||
|
|
||
| * [RESTORE](/sql-statements/sql-statement-restore.md) | ||
| * [SHOW BACKUPS](/sql-statements/sql-statement-show-backups.md) |
Oops, something went wrong.