forked from pingcap/docs
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
TiDB Binlog in k8s management documentation (pingcap#1440)
* Add maintain TiDB Binlog documentations to facilitate the Binlog management in k8s Signed-off-by: Aylei <rayingecho@gmail.com> * Update toc Signed-off-by: Aylei <rayingecho@gmail.com> * Fix broken link Signed-off-by: Aylei <rayingecho@gmail.com> * Apply suggestions from code review Co-Authored-By: Calvin Weng <wenghao@pingcap.com> * Address review comments Signed-off-by: Aylei <rayingecho@gmail.com> * Apply suggestions from code review Co-Authored-By: Calvin Weng <wenghao@pingcap.com> * Apply changes to dev folder Signed-off-by: Aylei <rayingecho@gmail.com> * Apply changes Signed-off-by: Aylei <rayingecho@gmail.com> * Apply suggestions from code review Co-Authored-By: Tennix <tennix@users.noreply.github.com> * Apply review comments to 3.0 * Update default configuration of drainer Signed-off-by: Aylei <rayingecho@gmail.com>
- Loading branch information
Showing
10 changed files
with
324 additions
and
26 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,112 @@ | ||
| --- | ||
| title: Maintain TiDB Binlog | ||
| summary: Learn how to maintain TiDB Binlog of a TiDB cluster in Kubernetes. | ||
| category: how-to | ||
| --- | ||
|
|
||
| # Maintain TiDB Binlog | ||
|
|
||
| This document describes how to maintain [TiDB Binlog](reference/tidb-binlog-overview.md) of a TiDB cluster in Kubernetes. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - [Deploy TiDB Operator](tidb-in-kubernetes/deploy/tidb-operator.md); | ||
| - [Install Helm](tidb-in-kubernetes/reference/tools/in-kubernetes.md#use-helm) and configure it with the official PingCAP chart. | ||
|
|
||
| ## Enable TiDB Binlog of a TiDB cluster | ||
|
|
||
| TiDB Binlog is disabled in the TiDB cluster by default. To create a TiDB cluster with TiDB Binlog enabled, or enable TiDB Binlog in an existing TiDB cluster: | ||
|
|
||
| 1. Modify the `values.yaml` file as described below: | ||
|
|
||
| * Set `binlog.pump.create` to `true`. | ||
| * Set `binlog.drainer.create` to `true`. | ||
| * Set `binlog.pump.storageClassName` and `binlog.drainer.storageClassName` to an available `storageClass` in your Kubernetes cluster. | ||
| * Set `binlog.drainer.destDBType` to your desired downstream storage as needed, which is explained in details below. | ||
|
|
||
| TiDB Binlog supports three types of downstream storage: | ||
|
|
||
| * PersistenceVolume: the default downstream storage. You can configure a large PV for `drainer` (by modifying `binlog.drainer.storage`) in this case. | ||
| * MySQL compatible databases: enabled by setting `binlog.drainer.destDBType` to `mysql`. Meanwhile, you must configure the address and credential of the target database in `binlog.drainer.mysql`. | ||
| * Apache Kafka: enabled by setting `binlog.drainer.destDBType` to `kafka`. Meanwhile, you must configure the zookeeper address and Kafka address of the target cluster in `binlog.drainer.kafka`. | ||
|
|
||
| 2. Create a new TiDB cluster or update an existing cluster: | ||
|
|
||
| * Create a new TiDB cluster with TiDB Binlog enabled: | ||
|
|
||
| {{< copyable "shell-regular" >}} | ||
|
|
||
| ```shell | ||
| helm install pingcap/tidb-cluster --name=<release-name> --namespace=<namespace> --version=<chart-version> -f <values-file> | ||
| ``` | ||
|
|
||
| * Update an existing TiDB cluster to enable TiDB Binlog: | ||
|
|
||
| {{< copyable "shell-regular" >}} | ||
|
|
||
| ```shell | ||
| helm upgrade <release-name> pingcap/tidb-cluster --version=<chart-version> -f <values-file> | ||
| ``` | ||
|
|
||
| ## Deploy multiple drainers | ||
|
|
||
| By default, only one downstream drainer is created. You can install the `tidb-drainer` Helm chart to deploy more drainers for a TiDB cluster, as described below: | ||
|
|
||
| 1. Make sure that the PingCAP Helm repository is up to date: | ||
|
|
||
| {{< copyable "shell-regular" >}} | ||
|
|
||
| ```shell | ||
| helm repo update | ||
| ``` | ||
|
|
||
| {{< copyable "shell-regular" >}} | ||
|
|
||
| ```shell | ||
| helm search tidb-drainer -l | ||
| ``` | ||
|
|
||
| 2. Get the default `values.yaml` file to facilitate customization: | ||
|
|
||
| ```shell | ||
| helm inspect values pingcap/tidb-drainer --version=<chart-version> > values.yaml | ||
| ``` | ||
|
|
||
| 3. Modify the `values.yaml` file to specify the source TiDB cluster and the downstream database of the drainer. Here is an example: | ||
|
|
||
| ```yaml | ||
| clusterName: example-tidb | ||
| clusterVersion: v3.0.0 | ||
| storageClassName: local-storage | ||
| storage: 10Gi | ||
| config: | | ||
| detect-interval = 10 | ||
| [syncer] | ||
| worker-count = 16 | ||
| txn-batch = 20 | ||
| disable-dispatch = false | ||
| ignore-schemas = "INFORMATION_SCHEMA,PERFORMANCE_SCHEMA,mysql" | ||
| safe-mode = false | ||
| db-type = "tidb" | ||
| [syncer.to] | ||
| host = "slave-tidb" | ||
| user = "root" | ||
| password = "" | ||
| port = 4000 | ||
| ``` | ||
|
|
||
| The `clusterName` and `clusterVersion` must match the desired source TiDB cluster. | ||
|
|
||
| For complete configuration details, refer to [TiDB Binlog Drainer Configurations in Kubernetes](tidb-in-kubernetes/reference/configuration/tidb-drainer.md). | ||
|
|
||
| 4. Deploy the drainer: | ||
|
|
||
| {{< copyable "shell-regular" >}} | ||
|
|
||
| ```shell | ||
| helm install pingcap/tidb-drainer --name=<release-name> --namespace=<namespace> --version=<chart-version> -f values.yaml | ||
| ``` | ||
|
|
||
| > **Note:** | ||
| > | ||
| > This chart must be installed to the same namespace as the source TiDB cluster. |
46 changes: 46 additions & 0 deletions
46
dev/tidb-in-kubernetes/reference/configuration/tidb-drainer.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| --- | ||
| title: TiDB Binlog Drainer Configurations in Kubernetes | ||
| summary: Learn the configurations of a TiDB Binlog Drainer in Kubernetes. | ||
| category: reference | ||
| --- | ||
|
|
||
| # TiDB Binlog Drainer Configurations in Kubernetes | ||
|
|
||
| This document introduces the configuration parameters for a TiDB Binlog drainer in Kubernetes. | ||
|
|
||
| ## Configuration parameters | ||
|
|
||
| The following table contains all configuration parameters available for the `tidb-drainer` chart. Refer to [Use Helm](tidb-in-kubernetes/reference/tools/in-kubernetes.md#use-helm) to learn how to configure these parameters. | ||
|
|
||
| | Parameter | Description | Default Value | | ||
| | :----- | :---- | :----- | | ||
| | `clusterName` | The name of the source TiDB cluster | `demo` | | ||
| | `clusterVersion` | The version of the source TiDB cluster | `v3.0.1` | | ||
| | `baseImage` | The base image of TiDB Binlog | `pingcap/tidb-binlog` | | ||
| | `imagePullPolicy` | The pulling policy of the image | `IfNotPresent` | | ||
| | `logLevel` | The log level of the drainer process | `info` | | ||
| | `storageClassName` | `storageClass` used by the drainer. `storageClassName` refers to a type of storage provided by the Kubernetes cluster, which might map to a level of service quality, a backup policy, or to any policy determined by the cluster administrator. Detailed reference: [storage-classes](https://kubernetes.io/docs/concepts/storage/storage-classes) | `local-storage` | | ||
| | `storage` | The storage limit of the drainer Pod. Note that you should set a larger size if `db-type` is set to `pb` | `10Gi` | | ||
| | `disableDetect` | Determines whether to disable casualty detection | `false` | | ||
| | `initialCommitTs` | Used to initialize a checkpoint if the drainer does not have one | `0` | | ||
| | `config` | The configuration file passed to the drainer. Detailed reference: [drainer.toml](https://github.com/pingcap/tidb-binlog/blob/master/cmd/drainer/drainer.toml) | (see below) | | ||
| | `resources` | The resource limits and requests of the drainer Pod | `{}` | | ||
| | `nodeSelector` | Ensures that the drainer Pod is only scheduled to the node with the specific key-value pair as the label. Detailed reference: [nodeselector](https://kubernetes.io/docs/concepts/configuration/assign-Pod-node/#nodeselector) | `{}` | | ||
| | `tolerations` | Applies to drainer Pods, allowing the Pods to be scheduled to the nodes with specified taints. Detailed reference: [taint-and-toleration](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration) | `{}` | | ||
| | `affinity` | Defines scheduling policies and preferences of the drainer Pod. Detailed reference:[affinity-and-anti-affinity](https://kubernetes.io/docs/concepts/configuration/assign-Pod-node/#affinity-and-anti-baffinity) | `{}` | | ||
|
|
||
| The default value of `config` is: | ||
|
|
||
| ```toml | ||
| detect-interval = 10 | ||
| compressor = "" | ||
| [syncer] | ||
| worker-count = 16 | ||
| disable-dispatch = false | ||
| ignore-schemas = "INFORMATION_SCHEMA,PERFORMANCE_SCHEMA,mysql" | ||
| safe-mode = false | ||
| txn-batch = 20 | ||
| db-type = "file" | ||
| [syncer.to] | ||
| dir = "/data/pb" | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,112 @@ | ||
| --- | ||
| title: Maintain TiDB Binlog | ||
| summary: Learn how to maintain TiDB Binlog of a TiDB cluster in Kubernetes. | ||
| category: how-to | ||
| --- | ||
|
|
||
| # Maintain TiDB Binlog | ||
|
|
||
| This document describes how to maintain [TiDB Binlog](reference/tidb-binlog-overview.md) of a TiDB cluster in Kubernetes. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - [Deploy TiDB Operator](tidb-in-kubernetes/deploy/tidb-operator.md); | ||
| - [Install Helm](tidb-in-kubernetes/reference/tools/in-kubernetes.md#use-helm) and configure it with the official PingCAP chart. | ||
|
|
||
| ## Enable TiDB Binlog of a TiDB cluster | ||
|
|
||
| TiDB Binlog is disabled in the TiDB cluster by default. To create a TiDB cluster with TiDB Binlog enabled, or enable TiDB Binlog in an existing TiDB cluster: | ||
|
|
||
| 1. Modify the `values.yaml` file as described below: | ||
|
|
||
| * Set `binlog.pump.create` to `true`. | ||
| * Set `binlog.drainer.create` to `true`. | ||
| * Set `binlog.pump.storageClassName` and `binlog.drainer.storageClassName` to an available `storageClass` in your Kubernetes cluster. | ||
| * Set `binlog.drainer.destDBType` to your desired downstream storage as needed, which is explained in details below. | ||
|
|
||
| TiDB Binlog supports three types of downstream storage: | ||
|
|
||
| * PersistenceVolume: the default downstream storage. You can configure a large PV for `drainer` (by modifying `binlog.drainer.storage`) in this case. | ||
| * MySQL compatible databases: enabled by setting `binlog.drainer.destDBType` to `mysql`. Meanwhile, you must configure the address and credential of the target database in `binlog.drainer.mysql`. | ||
| * Apache Kafka: enabled by setting `binlog.drainer.destDBType` to `kafka`. Meanwhile, you must configure the zookeeper address and Kafka address of the target cluster in `binlog.drainer.kafka`. | ||
|
|
||
| 2. Create a new TiDB cluster or update an existing cluster: | ||
|
|
||
| * Create a new TiDB cluster with TiDB Binlog enabled: | ||
|
|
||
| {{< copyable "shell-regular" >}} | ||
|
|
||
| ```shell | ||
| helm install pingcap/tidb-cluster --name=<release-name> --namespace=<namespace> --version=<chart-version> -f <values-file> | ||
| ``` | ||
|
|
||
| * Update an existing TiDB cluster to enable TiDB Binlog: | ||
|
|
||
| {{< copyable "shell-regular" >}} | ||
|
|
||
| ```shell | ||
| helm upgrade <release-name> pingcap/tidb-cluster --version=<chart-version> -f <values-file> | ||
| ``` | ||
|
|
||
| ## Deploy multiple drainers | ||
|
|
||
| By default, only one downstream drainer is created. You can install the `tidb-drainer` Helm chart to deploy more drainers for a TiDB cluster, as described below: | ||
|
|
||
| 1. Make sure that the PingCAP Helm repository is up to date: | ||
|
|
||
| {{< copyable "shell-regular" >}} | ||
|
|
||
| ```shell | ||
| helm repo update | ||
| ``` | ||
|
|
||
| {{< copyable "shell-regular" >}} | ||
|
|
||
| ```shell | ||
| helm search tidb-drainer -l | ||
| ``` | ||
|
|
||
| 2. Get the default `values.yaml` file to facilitate customization: | ||
|
|
||
| ```shell | ||
| helm inspect values pingcap/tidb-drainer --version=<chart-version> > values.yaml | ||
| ``` | ||
|
|
||
| 3. Modify the `values.yaml` file to specify the source TiDB cluster and the downstream database of the drainer. Here is an example: | ||
|
|
||
| ```yaml | ||
| clusterName: example-tidb | ||
| clusterVersion: v3.0.0 | ||
| storageClassName: local-storage | ||
| storage: 10Gi | ||
| config: | | ||
| detect-interval = 10 | ||
| [syncer] | ||
| worker-count = 16 | ||
| txn-batch = 20 | ||
| disable-dispatch = false | ||
| ignore-schemas = "INFORMATION_SCHEMA,PERFORMANCE_SCHEMA,mysql" | ||
| safe-mode = false | ||
| db-type = "tidb" | ||
| [syncer.to] | ||
| host = "slave-tidb" | ||
| user = "root" | ||
| password = "" | ||
| port = 4000 | ||
| ``` | ||
|
|
||
| The `clusterName` and `clusterVersion` must match the desired source TiDB cluster. | ||
|
|
||
| For complete configuration details, refer to [TiDB Binlog Drainer Configurations in Kubernetes](tidb-in-kubernetes/reference/configuration/tidb-drainer.md). | ||
|
|
||
| 4. Deploy the drainer: | ||
|
|
||
| {{< copyable "shell-regular" >}} | ||
|
|
||
| ```shell | ||
| helm install pingcap/tidb-drainer --name=<release-name> --namespace=<namespace> --version=<chart-version> -f values.yaml | ||
| ``` | ||
|
|
||
| > **Note:** | ||
| > | ||
| > This chart must be installed to the same namespace as the source TiDB cluster. |
Oops, something went wrong.