| title | summary | category |
|---|---|---|
Upgrade Loader or Syncer to Data Migration |
This document introduces how to upgrade Loader or Syncer to Data Migration. |
tools |
This document introduces how to upgrade Loader or Syncer to DM (Data Migration).
Loader is a tool used to load the full data that is dumped from mydumper to TiDB.
When the task-mode of the task DM executes is full, DM automatically uses dumper to dump data and then uses loader to load the data.
To upgrade Loader to DM, perform the following steps:
- Deploy the DM cluster.
- Refer to Loader configuration change and generate the corresponding task configuration file.
- Use dmctl to start the task.
Loader uses the TOML file to define the process-related operation parameters and synchronization task parameters; DM uses the YAML file to define task configuration file parameters.
Taking the configuration options in Data Migration Task Configuration File as examples, the corresponding relationship of task configuration options between Loader and DM is as follows:
| Configuration in Loader | Corresponding configuration in DM |
|---|---|
pool-size |
pool-size of loader |
dir |
dir of loader |
db |
target-database |
alternative-db |
Deprecated |
source-db |
Deprecated |
route-rules |
route-rules |
pattern-schema |
schema-pattern |
pattern-table |
table-pattern |
do-db |
do-dbs of black-white-list. The filtering feature of databases has been refactored. For detailed configuration, see do-dbs in Data Migration Task Configuration File. |
do-table |
do-tables of black-white-list. The filtering feature of tables has been refactored. For detailed configuration, see do-tables in Data Migration Task Configuration File. |
ignore-db |
ignore-dbs of black-white-list. The filtering feature of databases has been refactored. For detailed configuration, see ignore-dbs in Data Migration Task Configuration File. |
ignore-table |
ignore-tables of black-white-list. The filtering feature of tables has been refactored. For detailed configuration, see ignore-tables in Data Migration Task Configuration File. |
rm-checkpoint |
Deprecated. A configuration option with a similar feature is remove-meta. |
Taking the configuration options in Data Migration Task Configuration File as examples, the corresponding relationship of task configuration options between mydumper and DM is as follows:
| Configuration in mydumper | Corresponding configuration in DM |
|---|---|
host |
No corresponding option. Configured when DM-worker is deployed. |
port |
No corresponding option. Configured when DM-worker is deployed. |
user |
No corresponding option. Configured when DM-worker is deployed. |
password |
No corresponding option. Configured when DM-worker is deployed. |
threads |
threads of mydumper |
chunk-filesize |
chunk-filesize of mydumper |
skip-tz-utc |
skip-tz-utc of mydumper |
Other options are specified using extra-args of mydumper. Their usage in DM is the same as that in mydumper.
Syncer is a tool used to import data incrementally. The task Syncer executes is corresponding to the synchronization task with incremental task-mode in DM. The syncer processing unit feature of dm-worker in DM is corresponding to the Syncer feature.
To upgrade Syncer to DM, perform the following steps:
- Deploy the DM cluster.
- Refer to Syncer configuration change and generate the corresponding task configuration file.
- Use dmctl to start the task.
Syncer uses the TOML file to define the process-related operation parameters and synchronization task parameters; DM uses the YAML file to define task configuration file parameters.
Taking the configuration options in Data Migration Task Configuration File as examples, the corresponding relationship of task configuration options between Syncer and DM is as follows:
| Configuration in Syncer | Corresponding configuration in DM |
|---|---|
server-id |
Transferred to dm-worker.toml |
flavor |
Transferred to dm-worker.toml |
enable-gtid |
Transferred to dm-worker.toml |
auto-fix-gtid |
Transferred to dm-worker.toml |
meta |
meta of mysql-instances. binlog-name/binlog-pos of meta in Syncer corresponds to that of mysql-instances. |
persistent-dir |
Deprecated |
worker-count |
worker-count of syncer |
batch |
batch of syncer |
max-retry |
max-retry of syncer |
do-db |
do-dbs of black-white-list. The filtering feature of databases has been refactored. For detailed configuration, see do-dbs in Data Migration Task Configuration File. |
do-table |
do-tables of black-white-list. The filtering feature of tables has been refactored. For detailed configuration, see do-tables in Data Migration Task Configuration File. |
ignore-db |
ingore-dbs of black-white-list. The filtering feature of databases has been refactored. For detailed configuration, see ignore-dbs in Data Migration Task Configuration File. |
ignore-table |
ignore-tables of black-white-list. The filtering feature of tables has been refactored. For detailed configuration, see ignore-tables in Data Migration Task Configuration File. |
skip-ddls |
Deprecated. Use filters. |
skip-sqls |
Deprecated. Use filters. |
skip-events |
Deprecated. Use filters. |
skip-dmls |
Deprecated. Use filters. |
route-rules |
route-rules |
pattern-schema |
schema-pattern |
pattern-table |
table-pattern |
from |
config of mysql-instances. Keep it consistent with the upstream MySQL information during the DM-worker deployment. |
to |
target-database |
disable-detect |
disable-detect of syncer |
safe-mode |
safe-mode of syncer |
stop-on-ddl |
Deprecated |
execute-ddl-timeout |
Deprecated |
execute-dml-timeout |
Deprecated |