diff --git a/README.md b/README.md index 4deda3fbeaa88..3983b3745b9a8 100644 --- a/README.md +++ b/README.md @@ -123,30 +123,29 @@ - [mydumper](tools/mydumper.md) - [Loader](tools/loader.md) + Data Migration - - [Overview](tools/data-migration-overview.md) - - [Restrictions](tools/data-migration-overview.md#usage-restrictions) + - [Overview](/tools/dm/overview.md) + - [Restrictions](/tools/dm/overview.md#usage-restrictions) + - [Deploy](/tools/dm/practice.md) + Features - - [Table Routing](tools/dm-data-synchronization-features.md#table-routing) - - [Black and White Lists](tools/dm-data-synchronization-features.md#black-and-white-table-lists) - - [Binlog Event Filtering](tools/dm-data-synchronization-features.md#binlog-event-filtering) - - [Column Mapping](tools/dm-data-synchronization-features.md#column-mapping) - - [Synchronization Delay Monitoring](tools/dm-data-synchronization-features.md#synchronization-delay-monitoring) + - [Table Routing](/tools/dm/data-synchronization-features.md#table-routing) + - [Black and White Lists](/tools/dm/data-synchronization-features.md#black-and-white-table-lists) + - [Binlog Event Filtering](/tools/dm/data-synchronization-features.md#binlog-event-filtering) + - [Column Mapping](/tools/dm/data-synchronization-features.md#column-mapping) + - [Synchronization Delay Monitoring](/tools/dm/data-synchronization-features.md#synchronization-delay-monitoring) + Sharding Support - - [Introduction](tools/dm-sharding-solution.md) - - [Restrictions](tools/dm-sharding-solution.md#sharding-ddl-usage-restrictions) - - [Troubleshoot](tools/troubleshooting-sharding-ddl-locks.md) + - [Introduction](/tools/dm/sharding-solution.md) + - [Restrictions](/tools/dm/sharding-solution.md#sharding-ddl-usage-restrictions) + - [Troubleshoot](/tools/dm/troubleshooting-sharding-ddl-locks.md) + Usage Scenarios - - [Simple Scenario](tools/dm-simple-synchronization-scenario.md) - - [Shard Merge Scenario](tools/dm-shard-merge-scenario.md) - - [Deploy](tools/data-migration-practice.md) + - [Simple Scenario](/tools/dm/simple-synchronization-scenario.md) + - [Shard Merge Scenario](/tools/dm/shard-merge-scenario.md) + Configure - - [Overview](tools/dm-configuration-file-overview.md) - - [Task Configuration](tools/dm-task-configuration-file-intro.md) - - [Monitor](tools/dm-monitor.md) - - [Manage the Task](tools/data-migration-manage-task.md) - - [Cluster Operations](tools/data-migration-cluster-operations.md) - - [Upgrade Loader or Syncer to DM](tools/upgrade-loader-or-syncer-to-dm.md) - - [Troubleshoot](tools/data-migration-troubleshooting.md) + - [Overview](/tools/dm/dm-configuration-file-overview.md) + - [Task Configuration](/tools/dm/task-configuration-file-intro.md) + - [Monitor](/tools/dm/monitor.md) + - [Manage the Task](/tools/dm/manage-task.md) + - [Cluster Operations](/tools/dm/cluster-operations.md) + - [Troubleshoot](/tools/dm/troubleshooting.md) + TiDB-Lightning - [Overview](tools/lightning/overview-architecture.md) - [Deployment](tools/lightning/deployment.md) diff --git a/tools/data-migration-cluster-operations.md b/tools/dm/cluster-operations.md similarity index 92% rename from tools/data-migration-cluster-operations.md rename to tools/dm/cluster-operations.md index da32c110e30f9..a3718606cfa27 100644 --- a/tools/data-migration-cluster-operations.md +++ b/tools/dm/cluster-operations.md @@ -40,7 +40,7 @@ This sections describes the considerations that you need to know when you restar **In the process of full data loading:** -For the SQL files during full data import, DM uses the downstream database to record the checkpoint information. When DM-worker is restarted, it checks the checkpoint information and you can use the [`start-task` command](../tools/data-migration-practice.md#step-4-start-the-data-synchronization-task) to recover the data synchronization task automatically. +For the SQL files during full data import, DM uses the downstream database to record the checkpoint information. When DM-worker is restarted, it checks the checkpoint information and you can use the [`start-task` command](/tools/dm/practice.md#step-4-start-the-data-synchronization-task) to recover the data synchronization task automatically. **In the process of incremental data synchronization:** @@ -60,7 +60,7 @@ For the binlog during incremental data import, DM uses the downstream database t At this time, DM tries again to synchronize these DDL statements that are not skipped. However, the restarted DM-worker instances will be blocked at the position of the binlog event corresponding to the DDL binlog event, because the DM-worker instance that is not restarted has executed to the place after this DDL binlog event. - To resolve this issue, follow the steps described in [Troubleshooting Sharding DDL Locks](../tools/troubleshooting-sharding-ddl-locks.md#condition-two-a-dm-worker-restarts-or-is-unreachable-temporarily) + To resolve this issue, follow the steps described in [Troubleshooting Sharding DDL Locks](/tools/dm/troubleshooting-sharding-ddl-locks.md#condition-two-a-dm-worker-restarts-or-is-unreachable-temporarily) #### Restarting DM-master considerations @@ -71,7 +71,7 @@ The information maintained by DM-master includes the following two major types, When DM-master is restarted, it automatically requests the task information from each DM-worker instance and rebuilds the corresponding relationship between the task and DM-worker. However, at this time, DM-worker does not resend the sharding DDL information, so it might occur that the sharding DDL lock synchronization cannot be finished automatically because of the lost lock information. -To resolve this issue, follow the steps described in [Troubleshooting Sharding DDL Locks](../tools/troubleshooting-sharding-ddl-locks.md#condition-three-dm-master-restarts). +To resolve this issue, follow the steps described in [Troubleshooting Sharding DDL Locks](/tools/dm/troubleshooting-sharding-ddl-locks.md#condition-three-dm-master-restarts). #### Restarting dmctl considerations @@ -173,7 +173,7 @@ Assuming that you want to add a DM-worker instance on the `172.16.10.74` machine 1. Configure the SSH mutual trust and sudo rules on the Control Machine. - 1. Refer to [Configure the SSH mutual trust and sudo rules on the Control Machine](../tools/data-migration-deployment.md#step-5-configure-the-ssh-mutual-trust-and-sudo-rules-on-the-control-machine), log in to the Control Machine using the `tidb` user account and add `172.16.10.74` to the `[servers]` section of the `hosts.ini` file. + 1. Refer to [Configure the SSH mutual trust and sudo rules on the Control Machine](/tools/dm/deployment.md#step-5-configure-the-ssh-mutual-trust-and-sudo-rules-on-the-control-machine), log in to the Control Machine using the `tidb` user account and add `172.16.10.74` to the `[servers]` section of the `hosts.ini` file. ``` $ cd /home/tidb/dm-ansible @@ -267,7 +267,7 @@ Assuming that the `172.16.10.71` machine needs to be maintained or this machine 1. Configure the SSH mutual trust and sudo rules on the Control machine. - 1. Refer to [Configure the SSH mutual trust and sudo rules on the Control Machine](../tools/data-migration-deployment.md#step-5-configure-the-ssh-mutual-trust-and-sudo-rules-on-the-control-machine), log in to the Control Machine using the `tidb` user account, and add `172.16.10.80` to the `[servers]` section of the `hosts.ini` file. + 1. Refer to [Configure the SSH mutual trust and sudo rules on the Control Machine](/tools/dm/deployment.md#step-5-configure-the-ssh-mutual-trust-and-sudo-rules-on-the-control-machine), log in to the Control Machine using the `tidb` user account, and add `172.16.10.80` to the `[servers]` section of the `hosts.ini` file. ``` $ cd /home/tidb/dm-ansible @@ -327,7 +327,7 @@ Assuming that the `172.16.10.72` machine needs to be maintained or this machine 1. Configure the SSH mutual trust and sudo rules on the Control Machine. - 1. Refer to [Configure the SSH mutual trust and sudo rules on the Control Machine](../tools/data-migration-deployment.md#step-5-configure-the-ssh-mutual-trust-and-sudo-rules-on-the-control-machine), log in to the Control Machine using the `tidb` user account, and add `172.16.10.75` to the `[servers]` section of the `hosts.ini` file. + 1. Refer to [Configure the SSH mutual trust and sudo rules on the Control Machine](/tools/dm/deployment.md#step-5-configure-the-ssh-mutual-trust-and-sudo-rules-on-the-control-machine), log in to the Control Machine using the `tidb` user account, and add `172.16.10.75` to the `[servers]` section of the `hosts.ini` file. ``` $ cd /home/tidb/dm-ansible diff --git a/tools/dm-data-synchronization-features.md b/tools/dm/data-synchronization-features.md similarity index 96% rename from tools/dm-data-synchronization-features.md rename to tools/dm/data-synchronization-features.md index be031eb17a05b..5abde77ce649f 100644 --- a/tools/dm-data-synchronization-features.md +++ b/tools/dm/data-synchronization-features.md @@ -33,7 +33,7 @@ routes: ### Parameter explanation -DM synchronizes the upstream MySQL or MariaDB instance table that matches the [`schema-pattern`/`table-pattern` rule provided by Table selector](../tools/dm-table-selector.md) to the downstream `target-schema`/`target-table`. +DM synchronizes the upstream MySQL or MariaDB instance table that matches the [`schema-pattern`/`table-pattern` rule provided by Table selector](/tools/dm/table-selector.md) to the downstream `target-schema`/`target-table`. ### Usage examples @@ -221,7 +221,7 @@ filters: ### Parameter explanation -- [`schema-pattern`/`table-pattern`](../tools/dm-table-selector.md): the binlog events or DDL SQL statements of upstream MySQL or MariaDB instance tables that match `schema-pattern`/`table-pattern` are filtered by the rules below. +- [`schema-pattern`/`table-pattern`](/tools/dm/table-selector.md): the binlog events or DDL SQL statements of upstream MySQL or MariaDB instance tables that match `schema-pattern`/`table-pattern` are filtered by the rules below. - `events`: the binlog event array. @@ -349,7 +349,7 @@ column-mappings: ### Parameter explanation -- [`schema-pattern`/`table-pattern`](../tools/dm-table-selector.md): to execute column value modifying operations on the upstream MySQL or MariaDB instance tables that match the `schema-pattern`/`table-pattern` filtering rule. +- [`schema-pattern`/`table-pattern`](/tools/dm/table-selector.md): to execute column value modifying operations on the upstream MySQL or MariaDB instance tables that match the `schema-pattern`/`table-pattern` filtering rule. - `source-column`, `target-column`: to modify the value of the `source-column` column according to specified `expression` and assign the new value to `target-column`. - `expression`: the expression used to modify data. Currently, only the `partition id` built-in expression is supported. @@ -451,4 +451,4 @@ enable-heartbeat: true - DM-worker updates the `TS_slave_task` synchronization time after each synchronization task obtains the `dm_heartbeat`.`heartbeat` binlog. - DM-worker queries the current `TS_master` timestamp in the corresponding upstream MySQL or MariaDB `dm_heartbeat`.`heartbeat` tables every 10 seconds, and calculates `task_lag` = `TS_master` - `TS_slave_task` for each task. -See the `replicate lag` in the [binlog replication](../tools/dm-monitor.md#binlog-replication) processing unit of DM monitoring metrics. \ No newline at end of file +See the `replicate lag` in the [binlog replication](/tools/dm/monitor.md#binlog-replication) processing unit of DM monitoring metrics. \ No newline at end of file diff --git a/tools/data-migration-deployment.md b/tools/dm/deployment.md similarity index 98% rename from tools/data-migration-deployment.md rename to tools/dm/deployment.md index 6f38538c6383b..c6a408e96e4c3 100644 --- a/tools/data-migration-deployment.md +++ b/tools/dm/deployment.md @@ -181,7 +181,7 @@ You can choose one of the following two types of cluster topology according to y - [The cluster topology of a single DM-worker instance on each node](#option-1-use-the-cluster-topology-of-a-single-dm-worker-instance-on-each-node) - Generally, it is recommended to deploy one DM-worker instance on each node. Howerver, if the CPU and memory of your machine is much better than the required in [Hardware and Software Requirements](../op-guide/recommendation.md), and you have more than 2 disks in one node or the capacity of one SSD is larger than 2 TB, you can deploy no more than 2 DM-worker instances on a single node. + Generally, it is recommended to deploy one DM-worker instance on each node. Howerver, if the CPU and memory of your machine is much better than the required in [Hardware and Software Requirements](/op-guide/recommendation.md), and you have more than 2 disks in one node or the capacity of one SSD is larger than 2 TB, you can deploy no more than 2 DM-worker instances on a single node. - [The cluster topology of multiple DM-worker instances on each node](#option-2-use-the-cluster-topology-of-multiple-dm-worker-instances-on-each-node) diff --git a/tools/dm-configuration-file-overview.md b/tools/dm/dm-configuration-file-overview.md similarity index 86% rename from tools/dm-configuration-file-overview.md rename to tools/dm/dm-configuration-file-overview.md index ea2645bdc9482..b154bf3c0cabc 100644 --- a/tools/dm-configuration-file-overview.md +++ b/tools/dm/dm-configuration-file-overview.md @@ -10,7 +10,7 @@ This document gives an overview of configuration files of DM (Data Migration). ## DM process configuration files -- `inventory.ini`: The configuration file of deploying DM using DM-Ansible. You need to edit it based on your machine topology. For details, see [Edit the `inventory.ini` file to orchestrate the DM cluster](../tools/data-migration-deployment.md#step-7-edit-the-inventoryini-file-to-orchestrate-the-dm-cluster). +- `inventory.ini`: The configuration file of deploying DM using DM-Ansible. You need to edit it based on your machine topology. For details, see [Edit the `inventory.ini` file to orchestrate the DM cluster](/tools/dm/deployment.md#step-7-edit-the-inventoryini-file-to-orchestrate-the-dm-cluster). - `dm-master.toml`: The configuration file of running the DM-master process, including the topology information of the DM cluster and the corresponding relationship between the MySQL instance and DM-worker (must be one-to-one relationship). When you use DM-Ansible to deploy DM, `dm-master.toml` is generated automatically. - `dm-worker.toml`: The configuration file of running the DM-worker process, including the configuration information of upstream MySQL instance. When you use DM-Ansible to deploy DM, `dm-worker.toml` is generated automatically. @@ -20,14 +20,14 @@ This document gives an overview of configuration files of DM (Data Migration). When you use DM-Ansible to deploy DM, you can find the following task configuration file template in `/conf`: -- `task.yaml.exmaple`: The standard configuration file of the data synchronization task (a specific task corresponds to a `task.yaml`). For the introduction of the configuration file, see [Task Configuration File](../tools/dm-task-configuration-file-intro.md). +- `task.yaml.exmaple`: The standard configuration file of the data synchronization task (a specific task corresponds to a `task.yaml`). For the introduction of the configuration file, see [Task Configuration File](/tools/dm/task-configuration-file-intro.md). ### Data synchronization task creation You can perform the following steps to create a data synchronization task based on `task.yaml.example`: 1. Copy `task.yaml.example` as `your_task.yaml`. -2. Refer to the description in the [Task Configuration File](../tools/dm-task-configuration-file-intro.md) and modify the configuration in `your_task.yaml`. +2. Refer to the description in the [Task Configuration File](/tools/dm/task-configuration-file-intro.md) and modify the configuration in `your_task.yaml`. 3. Create your data synchronization task using dmctl. ### Important concepts diff --git a/tools/dm-worker-intro.md b/tools/dm/dm-worker-intro.md similarity index 100% rename from tools/dm-worker-intro.md rename to tools/dm/dm-worker-intro.md diff --git a/tools/data-migration-manage-task.md b/tools/dm/manage-task.md similarity index 97% rename from tools/data-migration-manage-task.md rename to tools/dm/manage-task.md index cddab262e6c1c..2df0ef8646093 100644 --- a/tools/data-migration-manage-task.md +++ b/tools/dm/manage-task.md @@ -6,7 +6,7 @@ category: tools # Manage the Data Synchronization Task -This document describes how to manage and maintain the data synchronization task using the [dmctl](../tools/data-migration-overview.md#dmctl) component. For the Data Migration cluster deployed using DM-Ansible, the dmctl binary file is in `dm-ansible/dmctl`. +This document describes how to manage and maintain the data synchronization task using the [dmctl](/tools/dm/overview.md#dmctl) component. For the Data Migration cluster deployed using DM-Ansible, the dmctl binary file is in `dm-ansible/dmctl`. ## dmctl basic usage @@ -113,7 +113,7 @@ The user of the upstream and downstream databases must have the corresponding re TiDB differs from MySQL in compatibility in the following aspects: - Does not support the foreign key - - [Character set compatibility differs](../sql/character-set-support.md) + - [Character set compatibility differs](/sql/character-set-support.md) + The consistency check on the upstream MySQL multiple-instance shards @@ -555,7 +555,7 @@ update-task [-w "127.0.0.1:10181"] ./task.yaml ## Manage the DDL locks -See [Troubleshooting Sharding DDL Locks](../tools/troubleshooting-sharding-ddl-locks.md). +See [Troubleshooting Sharding DDL Locks](/tools/dm/troubleshooting-sharding-ddl-locks.md). ## Refresh worker tasks diff --git a/tools/dm-monitor.md b/tools/dm/monitor.md similarity index 96% rename from tools/dm-monitor.md rename to tools/dm/monitor.md index d6be0cd2dd08f..71248035b14a7 100644 --- a/tools/dm-monitor.md +++ b/tools/dm/monitor.md @@ -6,7 +6,7 @@ category: operations # Data Migration Monitoring Metrics -If your DM cluster is deployed using DM-Ansible, the [monitoring system](../tools/data-migration-practice.md#step-7-monitor-the-task-and-check-logs) is also deployed at the same time. This document describes the monitoring metrics provided by DM-worker. +If your DM cluster is deployed using DM-Ansible, the [monitoring system](/tools/dm/practice.md#step-7-monitor-the-task-and-check-logs) is also deployed at the same time. This document describes the monitoring metrics provided by DM-worker. > **Note:** Currently, DM-master does not provide monitoring metrics yet. diff --git a/tools/data-migration-overview.md b/tools/dm/overview.md similarity index 62% rename from tools/data-migration-overview.md rename to tools/dm/overview.md index 8cb89b179be82..aa2aa76c6fb2c 100644 --- a/tools/data-migration-overview.md +++ b/tools/dm/overview.md @@ -12,7 +12,7 @@ Data Migration (DM) is an integrated data synchronization task management platfo The Data Migration tool includes three components: DM-master, DM-worker, and dmctl. -![Data Migration architecture](../media/dm-architecture.png) +![Data Migration architecture](/media/dm-architecture.png) ### DM-master @@ -33,7 +33,7 @@ DM-worker executes specific data synchronization tasks. - Orchestrating the operation of the data synchronization subtasks - Monitoring the running state of the data synchronization subtasks -After DM-worker is started, it automatically synchronizes the upstream binlog to the local configuration directory (the default synchronization directory is `/relay_log` if DM is deployed using `DM-Ansible`). For details about DM-worker, see [DM-worker Introduction](../tools/dm-worker-intro.md). For details about the relay log, see [Relay Log](../tools/dm-relay-log.md). +After DM-worker is started, it automatically synchronizes the upstream binlog to the local configuration directory (the default synchronization directory is `/relay_log` if DM is deployed using `DM-Ansible`). For details about DM-worker, see [DM-worker Introduction](/tools/dm/dm-worker-intro.md). For details about the relay log, see [Relay Log](/tools/dm/relay-log.md). ### dmctl @@ -50,23 +50,23 @@ This section describes the data synchronization features provided by the Data Mi ### Schema and table routing -The [schema and table routing](../tools/dm-data-synchronization-features.md#table-routing) feature means that DM can synchronize a certain table of the upstream MySQL or MariaDB instance to the specified table in the downstream, which can be used to merge or synchronize the sharding data. +The [schema and table routing](/tools/dm/data-synchronization-features.md#table-routing) feature means that DM can synchronize a certain table of the upstream MySQL or MariaDB instance to the specified table in the downstream, which can be used to merge or synchronize the sharding data. ### Black and white lists synchronization at the schema and table levels -The [black and white lists filtering rule](../tools/dm-data-synchronization-features.md#black-and-white-table-lists) of the upstream database instance tables is similar to MySQL `replication-rules-db`/`replication-rules-table`, which can be used to filter or only synchronize all operations of some databases or some tables. +The [black and white lists filtering rule](/tools/dm/data-synchronization-features.md#black-and-white-table-lists) of the upstream database instance tables is similar to MySQL `replication-rules-db`/`replication-rules-table`, which can be used to filter or only synchronize all operations of some databases or some tables. ### Binlog event filtering -[Binlog event filtering](../tools/dm-data-synchronization-features.md#binlog-event-filtering) is a more fine-grained filtering rule than the black and white lists filtering rule. You can use statements like `INSERT` or `TRUNCATE TABLE` to specify the binlog events of `schema/table` that you need to synchronize or filter out. +[Binlog event filtering](/tools/dm/data-synchronization-features.md#binlog-event-filtering) is a more fine-grained filtering rule than the black and white lists filtering rule. You can use statements like `INSERT` or `TRUNCATE TABLE` to specify the binlog events of `schema/table` that you need to synchronize or filter out. ### Column mapping -The [column mapping](../tools/dm-data-synchronization-features.md#column-mapping) feature means that the table column value can be modified according to the built-in expression specified by the user, which can be used to resolve the conflicts of the sharding auto-increment primary key IDs. +The [column mapping](/tools/dm/data-synchronization-features.md#column-mapping) feature means that the table column value can be modified according to the built-in expression specified by the user, which can be used to resolve the conflicts of the sharding auto-increment primary key IDs. ### Sharding support -DM supports merging the original sharded instances and tables into TiDB, but with [some restrictions](../tools/dm-sharding-solution.md#sharding-ddl-usage-restrictions). +DM supports merging the original sharded instances and tables into TiDB, but with [some restrictions](/tools/dm/sharding-solution.md#sharding-ddl-usage-restrictions). ## Usage restrictions @@ -77,22 +77,22 @@ Before using the DM tool, note the following restrictions: - 5.5 < MySQL version < 5.8 - MariaDB version >= 10.1.2 - Data Migration [prechecks the corresponding privileges and configuration automatically](../tools/data-migration-manage-task.md#precheck-the-upstream-mysql-instance-configuration) while starting the data synchronization task using dmctl. + Data Migration [prechecks the corresponding privileges and configuration automatically](/tools/dm/manage-task.md#precheck-the-upstream-mysql-instance-configuration) while starting the data synchronization task using dmctl. + DDL syntax - - Currently, TiDB is not compatible with all the DDL statements that MySQL supports. Because DM uses the TiDB parser to process DDL statements, it only supports the DDL syntax supported by the TiDB parser. For details, see [the DDL statements supported by TiDB](../sql/ddl.md). + - Currently, TiDB is not compatible with all the DDL statements that MySQL supports. Because DM uses the TiDB parser to process DDL statements, it only supports the DDL syntax supported by the TiDB parser. For details, see [the DDL statements supported by TiDB](/sql/ddl.md). - - DM reports an error when it encounters an incompatible DDL statement. To solve this error, you need to manually handle it using dmctl, either skipping this DDL statement or replacing it with a specified DDL statement(s). For details, see [Skip or replace abnormal SQL statements](../tools/data-migration-troubleshooting.md#skip-or-replace-abnormal-sql-statements). + - DM reports an error when it encounters an incompatible DDL statement. To solve this error, you need to manually handle it using dmctl, either skipping this DDL statement or replacing it with a specified DDL statement(s). For details, see [Skip or replace abnormal SQL statements](/tools/dm/troubleshooting.md#skip-or-replace-abnormal-sql-statements). + Sharding - - If conflict exists between sharded tables, *only columns with the auto increment primary key* encounter the conflict, and the *column type is bigint*, solve the conflict using [column mapping](../tools/dm-data-synchronization-features.md#column-mapping). Otherwise, data synchronization is not supported. Conflicting data can cover each other and cause data loss. + - If conflict exists between sharded tables, *only columns with the auto increment primary key* encounter the conflict, and the *column type is bigint*, solve the conflict using [column mapping](/tools/dm/data-synchronization-features.md#column-mapping). Otherwise, data synchronization is not supported. Conflicting data can cover each other and cause data loss. - - For other sharding restrictions, see [Sharding DDL usage restrictions](../tools/dm-sharding-solution.md#sharding-ddl-usage-restrictions). + - For other sharding restrictions, see [Sharding DDL usage restrictions](/tools/dm/sharding-solution.md#sharding-ddl-usage-restrictions). + Operations - - After DM-worker is restarted, the data synchronization task cannot be automatically restored. You need to manually run `start-task`. For details, see [Manage the Data Synchronization Task](../tools/data-migration-manage-task.md). + - After DM-worker is restarted, the data synchronization task cannot be automatically restored. You need to manually run `start-task`. For details, see [Manage the Data Synchronization Task](/tools/dm/manage-task.md). - - After DM-worker or DM-master is restarted, the DDL lock synchronization cannot be automatically restored in some conditions. You need to manually handle it. For details, see [Troubleshooting Sharding DDL Locks](../tools/troubleshooting-sharding-ddl-locks.md). \ No newline at end of file + - After DM-worker or DM-master is restarted, the DDL lock synchronization cannot be automatically restored in some conditions. You need to manually handle it. For details, see [Troubleshooting Sharding DDL Locks](/tools/dm/troubleshooting-sharding-ddl-locks.md). \ No newline at end of file diff --git a/tools/data-migration-practice.md b/tools/dm/practice.md similarity index 97% rename from tools/data-migration-practice.md rename to tools/dm/practice.md index dcbfd0fabda1f..7be4379e197c1 100644 --- a/tools/data-migration-practice.md +++ b/tools/dm/practice.md @@ -10,11 +10,11 @@ This guide shows how to synchronize data using the Data Migration (DM) tool. ## Step 1: Deploy the DM cluster -It is recommended to deploy the DM cluster using DM-Ansible. For detailed deployment, see [Deploy Data Migration Using DM-Ansible](../tools/data-migration-deployment.md). +It is recommended to deploy the DM cluster using DM-Ansible. For detailed deployment, see [Deploy Data Migration Using DM-Ansible](/tools/dm/deployment.md). > **Notes**: > -> - For database passwords in all the DM configuration files, use the passwords encrypted by `dmctl`. If a database password is empty, it is unnecessary to encrypt it. See [Encrypt the upstream MySQL user password using dmctl](../tools/data-migration-deployment.md#encrypt-the-upstream-mysql-user-password-using-dmctl). +> - For database passwords in all the DM configuration files, use the passwords encrypted by `dmctl`. If a database password is empty, it is unnecessary to encrypt it. See [Encrypt the upstream MySQL user password using dmctl](/tools/dm/deployment.md#encrypt-the-upstream-mysql-user-password-using-dmctl). > - The user of the upstream and downstream databases must have the corresponding read and write privileges. ## Step 2: Check the cluster information diff --git a/tools/dm-query-status.md b/tools/dm/query-status.md similarity index 99% rename from tools/dm-query-status.md rename to tools/dm/query-status.md index 60bb762f7d08b..4f2b7636d3049 100644 --- a/tools/dm-query-status.md +++ b/tools/dm/query-status.md @@ -182,7 +182,7 @@ This document introduces the query result and subtask status of Data Migration ( ``` For the status description and status switch relationship of "stage" of "subTaskStatus" of "workers", see [Subtask status](#subtask-status). -For operation details of "unresolvedDDLLockID" of "subTaskStatus" of "workers", see [Troubleshooting Sharding DDL Locks](../tools/troubleshooting-sharding-ddl-locks/). +For operation details of "unresolvedDDLLockID" of "subTaskStatus" of "workers", see [Troubleshooting Sharding DDL Locks](/tools/dm/troubleshooting-sharding-ddl-locks/). ## Subtask status diff --git a/tools/dm-relay-log.md b/tools/dm/relay-log.md similarity index 96% rename from tools/dm-relay-log.md rename to tools/dm/relay-log.md index 24b744f612810..9f7f09d81078a 100644 --- a/tools/dm-relay-log.md +++ b/tools/dm/relay-log.md @@ -36,7 +36,7 @@ An example of the directory structure of the local storage for a relay log: - `subdir` is named `.`. - - After [a switch between master and slave instances](../tools/data-migration-cluster-operations.md#switch-between-master-and-slave-instances) in the upstream, DM-worker generates a new `subdir` directory with an incremental serial number. + - After [a switch between master and slave instances](/tools/dm/cluster-operations.md#switch-between-master-and-slave-instances) in the upstream, DM-worker generates a new `subdir` directory with an incremental serial number. - In the above example, for the `7e427cc0-091c-11e9-9e45-72b7c59d52d7.000001` directory, `7e427cc0-091c-11e9-9e45-72b7c59d52d7` is the upstream database UUID and `000001` is the local `subdir` serial number. diff --git a/tools/dm-shard-merge-scenario.md b/tools/dm/shard-merge-scenario.md similarity index 92% rename from tools/dm-shard-merge-scenario.md rename to tools/dm/shard-merge-scenario.md index 0a1d039831817..535892e0d4ae2 100644 --- a/tools/dm-shard-merge-scenario.md +++ b/tools/dm/shard-merge-scenario.md @@ -58,7 +58,7 @@ Assume that the downstream schema after synchronization is as follows: ## Synchronization solution -- To satisfy the synchronization Requirements #1 and #2, configure the [table routing rule](../tools/dm-data-synchronization-features.md#table-routing) as follows: +- To satisfy the synchronization Requirements #1 and #2, configure the [table routing rule](/tools/dm/data-synchronization-features.md#table-routing) as follows: ```yaml routes: @@ -68,7 +68,7 @@ Assume that the downstream schema after synchronization is as follows: target-schema: "user" ``` -- To satisfy the synchronization Requirement #3, configure the [table routing rule](../tools/dm-data-synchronization-features.md#table-routing) as follows: +- To satisfy the synchronization Requirement #3, configure the [table routing rule](/tools/dm/data-synchronization-features.md#table-routing) as follows: ```yaml routes: @@ -83,7 +83,7 @@ Assume that the downstream schema after synchronization is as follows: target-table: "sale" ``` -- To satisfy the synchronization Requirements #4 and #5, configure the [binlog event filtering rule](../tools/dm-data-synchronization-features.md#binlog-event-filtering) as follows: +- To satisfy the synchronization Requirements #4 and #5, configure the [binlog event filtering rule](/tools/dm/data-synchronization-features.md#binlog-event-filtering) as follows: ```yaml filters: @@ -96,7 +96,7 @@ Assume that the downstream schema after synchronization is as follows: > **Note:** The synchronization Requirements #4, #5 and #7 indicate that all the deletion operations in the `user` schema are filtered out, so a schema level filtering rule is configured here. However, the deletion operations of future tables in the `user` schema will also be filtered out. -- To satisfy the synchronization Requirement #6, configure the [binlog event filtering rule](../tools/dm-data-synchronization-features.md#binlog-event-filtering) as follows: +- To satisfy the synchronization Requirement #6, configure the [binlog event filtering rule](/tools/dm/data-synchronization-features.md#binlog-event-filtering) as follows: ```yaml filters: @@ -112,7 +112,7 @@ Assume that the downstream schema after synchronization is as follows: action: Ignore ``` -- To satisfy the synchronization Requirement #7, configure the [black and white table lists](../tools/dm-data-synchronization-features.md#black-and-white-table-lists) as follows: +- To satisfy the synchronization Requirement #7, configure the [black and white table lists](/tools/dm/data-synchronization-features.md#black-and-white-table-lists) as follows: ```yaml black-white-list: @@ -122,7 +122,7 @@ Assume that the downstream schema after synchronization is as follows: tbl-name: "log_bak" ``` -- To satisfy the synchronization Requirement #8, configure the [column mapping rule](../tools/dm-data-synchronization-features.md#column-mapping) as follows: +- To satisfy the synchronization Requirement #8, configure the [column mapping rule](/tools/dm/data-synchronization-features.md#column-mapping) as follows: ```yaml column-mappings: @@ -151,7 +151,7 @@ Assume that the downstream schema after synchronization is as follows: ## Synchronization task configuration -The complete configuration of the synchronization task is shown as below. For more details, see [Data Migration Task Configuration File](../tools/dm-task-configuration-file-intro.md). +The complete configuration of the synchronization task is shown as below. For more details, see [Data Migration Task Configuration File](/tools/dm/task-configuration-file-intro.md). ```yaml name: "shard_merge" diff --git a/tools/dm-sharding-solution.md b/tools/dm/sharding-solution.md similarity index 93% rename from tools/dm-sharding-solution.md rename to tools/dm/sharding-solution.md index 5fb992c57de1d..1c903323665c7 100644 --- a/tools/dm-sharding-solution.md +++ b/tools/dm/sharding-solution.md @@ -8,17 +8,17 @@ category: tools This document introduces the sharding solution provided by Data Migration, its background, design details, and sharding DDL restrictions. -Data Migration supports merging the data of multiple sharded MySQL instances and tables into a single TiDB instance. Generally, Data Migration does it automatically and you need to do nothing. But when some abnormal conditions occur, you need to handle them manually. For details, see [Troubleshooting Sharding DDL Locks](../tools/troubleshooting-sharding-ddl-locks.md). +Data Migration supports merging the data of multiple sharded MySQL instances and tables into a single TiDB instance. Generally, Data Migration does it automatically and you need to do nothing. But when some abnormal conditions occur, you need to handle them manually. For details, see [Troubleshooting Sharding DDL Locks](/tools/dm/troubleshooting-sharding-ddl-locks.md). ## Background -Currently, [Syncer](../tools/syncer.md) adopts the `ROW` format of binlog that does not contain the column name. The `ROW` format of binlog has the nature of self-description on a single machine. In this case, only one binlog updates the target database, and the column values corresponding to the newly added or modified column can be determined according to the downstream table schema, so the DML statements can be constructed correctly. +Currently, [Syncer](/tools/syncer.md) adopts the `ROW` format of binlog that does not contain the column name. The `ROW` format of binlog has the nature of self-description on a single machine. In this case, only one binlog updates the target database, and the column values corresponding to the newly added or modified column can be determined according to the downstream table schema, so the DML statements can be constructed correctly. However, in the case of merging multiple sharded instances and tables, multiple binlogs update the target database, and the DDL statements cannot be completely updated synchronously, so the DML statements might be inconsistent with the table schema. Here is an example: -![img](../media/shard-merge.png) +![img](/media/shard-merge.png) Assume that the table schema is as below: diff --git a/tools/dm-simple-synchronization-scenario.md b/tools/dm/simple-synchronization-scenario.md similarity index 90% rename from tools/dm-simple-synchronization-scenario.md rename to tools/dm/simple-synchronization-scenario.md index eb8c32998233d..b8cefec3ea8d7 100644 --- a/tools/dm-simple-synchronization-scenario.md +++ b/tools/dm/simple-synchronization-scenario.md @@ -61,7 +61,7 @@ Assume that the schemas synchronized to the downstream are as follows: ## Synchronization solution -- To satisfy synchronization Requirements #1-i, #1-ii and #1-iii, configure the [table routing rules](../tools/dm-data-synchronization-features.md#table-routing) as follows: +- To satisfy synchronization Requirements #1-i, #1-ii and #1-iii, configure the [table routing rules](/tools/dm/data-synchronization-features.md#table-routing) as follows: ```yaml routes: @@ -77,7 +77,7 @@ Assume that the schemas synchronized to the downstream are as follows: target-schema: "user_south" ``` -- To satisfy the synchronization Requirement #2-i, configure the [table routing rules](../tools/dm-data-synchronization-features.md#table-routing) as follows: +- To satisfy the synchronization Requirement #2-i, configure the [table routing rules](/tools/dm/data-synchronization-features.md#table-routing) as follows: ```yaml routes: @@ -94,7 +94,7 @@ Assume that the schemas synchronized to the downstream are as follows: target-table: "store_shenzhen" ``` -- To satisfy the synchronization Requirement #1-iv, configure the [binlog filtering rules](../tools/dm-data-synchronization-features.md#binlog-event-filtering) as follows: +- To satisfy the synchronization Requirement #1-iv, configure the [binlog filtering rules](/tools/dm/data-synchronization-features.md#binlog-event-filtering) as follows: ```yaml filters: @@ -110,7 +110,7 @@ Assume that the schemas synchronized to the downstream are as follows: action: Ignore ``` -- To satisfy the synchronization Requirement #2-ii, configure the [binlog filtering rule](../tools/dm-data-synchronization-features.md#binlog-event-filtering) as follows: +- To satisfy the synchronization Requirement #2-ii, configure the [binlog filtering rule](/tools/dm/data-synchronization-features.md#binlog-event-filtering) as follows: ```yaml filters: @@ -123,7 +123,7 @@ Assume that the schemas synchronized to the downstream are as follows: > **Note:** `store-filter-rule` is different from `log-filter-rule & user-filter-rule`. `store-filter-rule` is a rule for the whole `store` schema, while `log-filter-rule` and `user-filter-rule` are rules for the `log` table in the `user` schema. -- To satisfy the synchronization Requirement #3, configure the [black and white lists](../tools/dm-data-synchronization-features.md#black-and-white-table-lists) as follows: +- To satisfy the synchronization Requirement #3, configure the [black and white lists](/tools/dm/data-synchronization-features.md#black-and-white-table-lists) as follows: ```yaml black-white-list: @@ -133,7 +133,7 @@ Assume that the schemas synchronized to the downstream are as follows: ## Synchronization task configuration -The complete synchronization task configuration is shown below. For more details, see [configuration explanations](../tools/dm-task-configuration-file-intro.md). +The complete synchronization task configuration is shown below. For more details, see [configuration explanations](/tools/dm/task-configuration-file-intro.md). ```yaml name: "one-tidb-slave" diff --git a/tools/dm-table-selector.md b/tools/dm/table-selector.md similarity index 100% rename from tools/dm-table-selector.md rename to tools/dm/table-selector.md diff --git a/tools/dm-task-configuration-file-intro.md b/tools/dm/task-configuration-file-intro.md similarity index 94% rename from tools/dm-task-configuration-file-intro.md rename to tools/dm/task-configuration-file-intro.md index f819de2eb7498..12217d461d3c9 100644 --- a/tools/dm-task-configuration-file-intro.md +++ b/tools/dm/task-configuration-file-intro.md @@ -7,13 +7,13 @@ category: tools # Data Migration Task Configuration File This document introduces the task configuration file of Data Migration -- -[`task.yaml`](https://github.com/pingcap/tidb-tools/blob/docs/docs/dm/zh_CN/configuration/task.yaml), including [Global configuration](#global-configuration) and [Instance configuration](#instance-configuration). +[`task.yaml`](https://github.com/pingcap/dm/blob/master/dm/master/task.yaml), including [Global configuration](#global-configuration) and [Instance configuration](#instance-configuration). -For the feature and configuration of each configuration item, see [Data Synchronization Features](../tools/dm-data-synchronization-features.md). +For the feature and configuration of each configuration item, see [Data Synchronization Features](/tools/dm/data-synchronization-features.md). ## Important concepts -For description of important concepts including `instance-id` and the DM-worker ID, see [Important concepts](../tools/dm-configuration-file-overview.md#important-concepts). +For description of important concepts including `instance-id` and the DM-worker ID, see [Important concepts](/tools/dm/dm-configuration-file-overview.md#important-concepts). ## Global configuration diff --git a/tools/troubleshooting-sharding-ddl-locks.md b/tools/dm/troubleshooting-sharding-ddl-locks.md similarity index 100% rename from tools/troubleshooting-sharding-ddl-locks.md rename to tools/dm/troubleshooting-sharding-ddl-locks.md diff --git a/tools/data-migration-troubleshooting.md b/tools/dm/troubleshooting.md similarity index 93% rename from tools/data-migration-troubleshooting.md rename to tools/dm/troubleshooting.md index b009977410ff6..30dd74b1116f9 100644 --- a/tools/data-migration-troubleshooting.md +++ b/tools/dm/troubleshooting.md @@ -26,9 +26,9 @@ However, you need to reset the data synchronization task in some cases. For deta ### `Access denied for user 'root'@'172.31.43.27' (using password: YES)` shows when you query the task or check the log -For database related passwords in all the DM configuration files, use the passwords encrypted by `dmctl`. If a database password is empty, it is unnecessary to encrypt it. For how to encrypt the plaintext password, see [Encrypt the upstream MySQL user password using dmctl](../tools/data-migration-deployment.md#encrypt-the-upstream-mysql-user-password-using-dmctl). +For database related passwords in all the DM configuration files, use the passwords encrypted by `dmctl`. If a database password is empty, it is unnecessary to encrypt it. For how to encrypt the plaintext password, see [Encrypt the upstream MySQL user password using dmctl](/tools/dm/deployment.md#encrypt-the-upstream-mysql-user-password-using-dmctl). -In addition, the user of the upstream and downstream databases must have the corresponding read and write privileges. Data Migration also [prechecks the corresponding privileges automatically](../tools/data-migration-manage-task.md#precheck-the-upstream-mysql-instance-configuration) while starting the data synchronization task. +In addition, the user of the upstream and downstream databases must have the corresponding read and write privileges. Data Migration also [prechecks the corresponding privileges automatically](/tools/dm/manage-task.md#precheck-the-upstream-mysql-instance-configuration) while starting the data synchronization task. ### Incompatible DDL statements @@ -43,7 +43,7 @@ encountered incompatible DDL in TiDB: %s if the DDL is not needed, you can use dm-ctl to skip it, otherwise u also can use dm-ctl to replace it. ``` -> **Note:** Currently, TiDB is not compatible with all the DDL statements that MySQL supports. See [the DDL statements supported by TiDB](../sql/ddl.md). +> **Note:** Currently, TiDB is not compatible with all the DDL statements that MySQL supports. See [the DDL statements supported by TiDB](/sql/ddl.md). ## Reset the data synchronization task @@ -55,14 +55,14 @@ You need to reset the entire data synchronization task in the following cases: Generally, at this time, the relay unit exits with an error and cannot be automatically restored gracefully. You need to manually restore the data synchronization and the steps are as follows: 1. Use the `stop-task` command to stop all the synchronization tasks that are currently running. -2. Use Ansible to [stop the entire DM cluster](../tools/data-migration-deployment.md#step-10-stop-the-dm-cluster). +2. Use Ansible to [stop the entire DM cluster](/tools/dm/deployment.md#step-10-stop-the-dm-cluster). 3. Manually clean up the relay log directory of the DM-worker corresponding to the MySQL master whose binlog is reset. - If the cluster is deployed using DM-Ansible, the relay log is in the `/relay_log` directory. - If the cluster is manually deployed using the binary, the relay log is in the directory set in the `relay-dir` parameter. 4. Clean up downstream synchronized data. -5. Use Ansible to [start the entire DM cluster](../tools/data-migration-deployment.md#step-9-deploy-the-dm-cluster). +5. Use Ansible to [start the entire DM cluster](/tools/dm/deployment.md#step-9-deploy-the-dm-cluster). 6. Restart data synchronization with the new task name, or set `remove-meta` to `true` and `task-mode` to `all`. ## Skip or replace abnormal SQL statements diff --git a/tools/upgrade-loader-or-syncer-to-dm.md b/tools/upgrade-loader-or-syncer-to-dm.md deleted file mode 100644 index 29efd979af5e2..0000000000000 --- a/tools/upgrade-loader-or-syncer-to-dm.md +++ /dev/null @@ -1,103 +0,0 @@ ---- -title: Upgrade Loader or Syncer to Data Migration -summary: This document introduces how to upgrade Loader or Syncer to Data Migration. -category: tools ---- - -# Upgrade Loader or Syncer to Data Migration - -This document introduces how to upgrade Loader or Syncer to DM (Data Migration). - -## Upgrade Loader to Data Migration - -Loader is a tool used to load the full data that is dumped from [mydumper](../tools/mydumper.md) to TiDB. - -When the `task-mode` of the task DM executes is `full`, DM automatically uses `dumper` to dump data and then uses `loader` to load the data. - -To upgrade Loader to DM, perform the following steps: - -1. [Deploy the DM cluster](../tools/data-migration-deployment.md). -2. Refer to [Loader configuration change](#loader-configuration-change) and generate the corresponding task configuration file. -3. [Use dmctl to start the task](../tools/data-migration-manage-task.md#create-the-data-synchronization-task). - -### Loader Configuration change - -Loader uses the TOML file to define the process-related operation parameters and synchronization task parameters; DM uses the YAML file to define task configuration file parameters. - -Taking the configuration options in [Data Migration Task Configuration File](../tools/dm-task-configuration-file-intro.md) as examples, the corresponding relationship of task configuration options between Loader and DM is as follows: - -| Configuration in Loader | Corresponding configuration in DM | -| ---------------------- | --------------------------------- | -| `pool-size` | `pool-size` of `loader` | -| `dir` | `dir` of `loader` | -| `db` | `target-database` | -| `alternative-db` | Deprecated | -| `source-db` | Deprecated | -| `route-rules` | `route-rules` | -| `pattern-schema` | `schema-pattern` | -| `pattern-table` | `table-pattern` | -| `do-db` | `do-dbs` of `black-white-list`. The filtering feature of databases has been refactored. For detailed configuration, see `do-dbs` in [Data Migration Task Configuration File](../tools/dm-task-configuration-file-intro.md). | -| `do-table` | `do-tables` of `black-white-list`. The filtering feature of tables has been refactored. For detailed configuration, see `do-tables` in [Data Migration Task Configuration File](../tools/dm-task-configuration-file-intro.md). | -| `ignore-db` | `ignore-dbs` of `black-white-list`. The filtering feature of databases has been refactored. For detailed configuration, see `ignore-dbs` in [Data Migration Task Configuration File](../tools/dm-task-configuration-file-intro.md). | -| `ignore-table` | `ignore-tables` of `black-white-list`. The filtering feature of tables has been refactored. For detailed configuration, see `ignore-tables` in [Data Migration Task Configuration File](../tools/dm-task-configuration-file-intro.md). | -| `rm-checkpoint` | Deprecated. A configuration option with a similar feature is `remove-meta`. | - -Taking the configuration options in [Data Migration Task Configuration File](../tools/dm-task-configuration-file-intro.md) as examples, the corresponding relationship of task configuration options between mydumper and DM is as follows: - -| Configuration in mydumper | Corresponding configuration in DM | -| ---------------------- | --------------------------------- | -| `host` | No corresponding option. Configured when DM-worker is deployed. | -| `port` | No corresponding option. Configured when DM-worker is deployed. | -| `user` | No corresponding option. Configured when DM-worker is deployed. | -| `password` | No corresponding option. Configured when DM-worker is deployed. | -| `threads` | `threads` of `mydumper` | -| `chunk-filesize` | `chunk-filesize` of `mydumper` | -| `skip-tz-utc` | `skip-tz-utc` of `mydumper` | - -Other options are specified using `extra-args` of `mydumper`. Their usage in DM is the same as that in mydumper. - -## Upgrade Syncer to Data Migration - -Syncer is a tool used to import data incrementally. The task Syncer executes is corresponding to the synchronization task with `incremental` `task-mode` in DM. The `syncer` processing unit feature of dm-worker in DM is corresponding to the Syncer feature. - -To upgrade Syncer to DM, perform the following steps: - -1. [Deploy the DM cluster](../tools/data-migration-deployment.md). -2. Refer to [Syncer configuration change](#syncer-configuration-change) and generate the corresponding task configuration file. -3. [Use dmctl to start the task](../tools/data-migration-manage-task.md#create-the-data-synchronization-task). - -### Syncer Configuration change - -Syncer uses the TOML file to define the process-related operation parameters and synchronization task parameters; DM uses the YAML file to define task configuration file parameters. - -Taking the configuration options in [Data Migration Task Configuration File](../tools/dm-task-configuration-file-intro.md) as examples, the corresponding relationship of task configuration options between Syncer and DM is as follows: - -| Configuration in Syncer | Corresponding configuration in DM | -| ---------------------- | --------------------------------- | -| `server-id` | Transferred to `dm-worker.toml` | -| `flavor` | Transferred to `dm-worker.toml` | -| `enable-gtid` | Transferred to `dm-worker.toml` | -| `auto-fix-gtid` | Transferred to `dm-worker.toml` | -| `meta` | `meta` of `mysql-instances`. `binlog-name`/`binlog-pos` of `meta` in Syncer corresponds to that of `mysql-instances`. | -| `persistent-dir` | Deprecated | -| `worker-count` | `worker-count` of `syncer` | -| `batch` | `batch` of `syncer` | -| `max-retry` | `max-retry` of `syncer` | -| `do-db` | `do-dbs` of `black-white-list`. The filtering feature of databases has been refactored. For detailed configuration, see `do-dbs` in [Data Migration Task Configuration File](../tools/dm-task-configuration-file-intro.md). | -| `do-table` | `do-tables` of `black-white-list`. The filtering feature of tables has been refactored. For detailed configuration, see `do-tables` in [Data Migration Task Configuration File](../tools/dm-task-configuration-file-intro.md). | -| `ignore-db` | `ingore-dbs` of `black-white-list`. The filtering feature of databases has been refactored. For detailed configuration, see `ignore-dbs` in [Data Migration Task Configuration File](../tools/dm-task-configuration-file-intro.md). | -| `ignore-table` | `ignore-tables` of `black-white-list`. The filtering feature of tables has been refactored. For detailed configuration, see `ignore-tables` in [Data Migration Task Configuration File](../tools/dm-task-configuration-file-intro.md). | -| `skip-ddls` | Deprecated. Use `filters`. | -| `skip-sqls` | Deprecated. Use `filters`. | -| `skip-events` | Deprecated. Use `filters`. | -| `skip-dmls` | Deprecated. Use `filters`. | -| `route-rules` | `route-rules`| -| `pattern-schema` | `schema-pattern` | -| `pattern-table` | `table-pattern` | -| `from` | `config` of `mysql-instances`. Keep it consistent with the upstream MySQL information during the DM-worker deployment. | -| `to` | `target-database` | -| `disable-detect` | `disable-detect` of `syncer` | -| `safe-mode` | `safe-mode` of `syncer` | -| `stop-on-ddl` | Deprecated | -| `execute-ddl-timeout` | Deprecated | -| `execute-dml-timeout` | Deprecated |