Skip to content

Commit

Permalink
tiflash: add docs (pingcap#3850)
Browse files Browse the repository at this point in the history
* tiflash: update docs

* Update encryption-at-rest.md

* Update encryption-at-rest.md

* improve language

* fix errors

* Apply suggestions from code review

Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com>

* Update encryption-at-rest.md

Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com>
  • Loading branch information
Joyinqin and TomShawn authored Sep 9, 2020
1 parent 323a643 commit 2ec618f
Show file tree
Hide file tree
Showing 2 changed files with 32 additions and 9 deletions.
25 changes: 21 additions & 4 deletions enable-tls-between-components.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,9 +69,9 @@ Currently, it is not supported to only enable encrypted transmission of some spe
key-path = "/path/to/pd-server-key.pem"
```

- TiFlash
- TiFlash (New in v4.0.5)

Configure in the configuration file, and change the `http_port` item to `https_port`:
Configure in the `tiflash.toml` file, and change the `http_port` item to `https_port`:

```toml
[security]
Expand All @@ -83,6 +83,16 @@ Currently, it is not supported to only enable encrypted transmission of some spe
key_path = "/path/to/tiflash-server-key.pem"
```

Configure in the `tiflash-learner.toml` file:

```toml
[security]
# Sets the path for certificates. The empty string means that secure connections are disabled.
ca-path = "/path/to/ca.pem"
cert-path = "/path/to/tiflash-server.pem"
key-path = "/path/to/tiflash-server-key.pem"
```

- TiCDC

Configure in the command-line arguments and set the corresponding URL to `https`:
Expand Down Expand Up @@ -165,15 +175,22 @@ To verify component caller's identity, you need to mark the certificate user ide
cdc server --pd=https://127.0.0.1:2379 --log-file=ticdc.log --addr=0.0.0.0:8301 --advertise-addr=127.0.0.1:8301 --ca=/path/to/ca.pem --cert=/path/to/ticdc-cert.pem --key=/path/to/ticdc-key.pem --cert-allowed-cn="client1,client2"
```

- TiFlash
- TiFlash (New in v4.0.5)

Configure in the configuration file or command-line arguments:
Configure in the `tiflash.toml` file or command-line arguments:

```toml
[security]
cert_allowed_cn = ["TiKV-Server", "TiDB-Server"]
```

Configure in the `tiflash-learner.toml` file:

```toml
[security]
cert-allowed-cn = ["PD-Server", "TiKV-Server", "TiFlash-Server"]
```

### Reload certificates

To reload the certificates and the keys, TiDB, PD, TiKV, and all kinds of clients reread the current certificates and the key files each time a new connection is created. Currently, you cannot reload the CA certificate.
Expand Down
16 changes: 11 additions & 5 deletions encryption-at-rest.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Also from v4.0.0, BR supports S3 server-side encryption (SSE) when backing up to
The current version of TiKV encryption has some drawbacks that we are working actively to address in the future versions.

* When a TiDB cluster is deployed, the majority of user data is stored in TiKV nodes, and that data will be encrypted when encryption is enabled. However, a small amount of user data is stored in PD nodes as metadata (for example, secondary index keys used as TiKV region boundaries). As of v4.0.0, PD doesn't support encryption at rest. It is recommended to use storage-level encryption (for example, file system encryption) to help protect sensitive data stored in PD.
* As of v4.0.0, TiFlash doesn't support encryption at rest. When deploying TiKV with TiFlash, data stored in TiFlash is not encrypted.
* TiFlash supports encryption at rest since v4.0.5. For details, refer to [Encryption at Rest for TiFlash](#encryption-at-rest-for-tiflash-new-in-v405). When deploying TiKV with TiFlash earlier than v4.0.5, data stored in TiFlash is not encrypted.
* TiKV currently does not exclude encryption keys and user data from core dumps. It is advised to disable core dumps for the TiKV process when using encryption at rest. This is not currently handled by TiKV itself.
* TiKV tracks encrypted data files using the absolute path of the files. As a result, once encryption is turned on for a TiKV node, the user should not change data file paths config such as `storage.data-dir`, `raftstore.raftdb-path`, `rocksdb.wal-dir` and `raftdb.wal-dir`.
* TiKV info log contains user data for debugging purposes. The info log and this data in it are not encrypted.
Expand All @@ -33,13 +33,13 @@ TiKV currently supports encrypting data using AES128, AES192 or AES256, in CTR m

The same master key can be shared by multiple instances of TiKV. The recommended way to provide a master key in production is via AWS KMS. Create a customer master key (CMK) through AWS KMS, and then provide the CMK key ID to TiKV in the config file. The TiKV process needs access to the KMS CMK while it is running, which can be done by using an [IAM role](https://aws.amazon.com/iam/). If TiKV fails to get access to the KMS CMK, it will fail to start or restart. Refer to AWS documentation for [KMS](https://docs.aws.amazon.com/kms/index.html) and [IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) usage.

Alternatively, if using custom key is desired, supplying the master key via file is also supported. The file needs to contain a 256 bits (or 32 bytes) key encoded as hex string. The file should end with a newline (i.e. `\n`) and contain nothing else. Persisting the key on disk, however, leaks the key, so the key file is only suitable to be stored on tempfs in RAM.
Alternatively, if using custom key is desired, supplying the master key via file is also supported. The file needs to contain a 256 bits (or 32 bytes) key encoded as hex string. The file should end with a newline (i.e. `\n`) and contain nothing else. Persisting the key on disk, however, leaks the key, so the key file is only suitable to be stored on the `tempfs` in RAM.

Data keys are generated by TiKV and passed to the underlying storage engine (i.e. RocksDB). All files written by RocksDB, including SST files, WAL files, and the MANIFEST file, are encrypted by the current data key. Other temporary files used by TiKV that may include user data are also encrypted using the same data key. Data keys are automatically rotated by TiKV every week by default, but the period is configurable. On key rotation, TiKV does not rewrite all existing files to replace the key, but RocksDB compaction are expected to rewrite old data into new data files, with the most recent data key, if the cluster gets constant write workload. TiKV keeps track of the key and encryption method used to encrypt each of the files and use the information to decrypt the content on reads.

Regardless of data encryption method, data keys are encrypted using AES256 in GCM mode for additional authentication. This required the master key to be 256 bits (32 bytes), when passing from file instead of KMS.

### Configuring Encryption
### Configuring encryption

To enable encryption, you can add the encryption section in TiKV's config file:

Expand Down Expand Up @@ -77,7 +77,7 @@ Here `path` is the path to the key file. The file must contain a 256 bits (or 16
3b5896b5be691006e0f71c3040a29495ddcad20b14aff61806940ebd780d3c62
```

### Rotating the Master Key
### Rotating the master key

To rotate master key, you have to specify both of the new master key and old master key in the config, and restart TiKV. Use `security.encryption.master-key` to specify the new master key, and use `security.encryption.previous-master-key` to specify the old master key. The config format for `security.encryption.previous-master-key` is the same as `encryption.master-key`. On restart TiKV must access both of the old and new master key, but once TiKV is up and running, TiKV will only need access to the new key. It is okay to leave the `encryption.previous-master-key` config in the config file from that on. Even on restart, TiKV only tries to use the old key if it fails to decrypt existing data using the new master key.

Expand All @@ -97,7 +97,7 @@ key-id = "0987dcba-09fe-87dc-65ba-ab0987654321"
region = "us-west-2"
```

### Monitoring and Debugging
### Monitoring and debugging

To monitor encryption at rest, if you deploy TiKV with Grafana, you can look at the **Encryption** panel in the **TiKV-Details** dashboard. There are a few metrics to look for:

Expand Down Expand Up @@ -128,3 +128,9 @@ When restoring the backup, both `--s3.sse` and `--s3.sse-kms-key-id` should NOT
```
./br restore full --pd <pd-address> --storage "s3://<bucket>/<prefix> --s3.region <region>"
```

## Encryption at rest for TiFlash <span class="version-mark">New in v4.0.5</span>

TiFlash supports encryption at rest since v4.0.5. Data keys are generated by TiFlash. All files (including data files, schema files, and temporary files) written into TiFlash (including TiFlash Proxy) are encrypted using the current data key. The encryption algorithms, the encryption configuration (in the `tiflash-learner.toml` file) supported by TiFlash, and the meanings of monitoring metrics are consistent with those of TiKV.

If you have deployed TiFlash with Grafana, you can check the **TiFlash-Proxy-Details** -> **Encryption** panel.

0 comments on commit 2ec618f

Please sign in to comment.