English | 中文

Introduction

• ape-dts is a data migration tool enabling any-to-any data transfers.
• It also provides data subscription and data processing.
• It is lightweight, efficient and standalone, requiring no third-party components or extra storage.
• Designed for cloud-native stateless component scenarios.
• In Rust.

Key features

• Supports data migration between various databases, both homogeneous and heterogeneous.
• Supports snapshot and cdc tasks with resume from breakpoint.
• Supports checking and revising data.
• Supports filtering and routing at the database, table, and column levels.
• Implements different parallel algorithms for different sources, targets, and task types to improve performance.
• Allows loading user-defined Lua scripts to modify the data.

Supported task types

	mysql -> mysql	pg -> pg	mongo -> mongo	redis -> redis	mysql -> kafka	pg -> kafka	mysql -> starrocks	mysql -> clickhouse	mysql -> tidb	pg -> starrocks	pg -> clickhouse	mysql -> doris	pg -> doris
Snapshot	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔
CDC	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔
Data check/revise/review	✔	✔	✔						✔
Structure migration	✔	✔					✔	✔	✔	✔	✔	✔	✔

Advanced

Crate features

The dt-main crate provides several optional components which can be enabled via Cargo [features]:

• metrics: Enable Prometheus format task metrics HTTP service interface. See the task metrics reference for metric names, units, and semantics. After enabling this feature, you can customize the metrics service with the following configuration:

  [metrics]
  # http service host
  http_host=127.0.0.1
  # http service port
  http_port=9090
  # http service worker count
  workers=2
  # prometheus metrics const labels
  labels=your_label1:your_value1,your_label2:your_value2
  

• TBD

Quick starts

CLI

dtscli is a lightweight local CLI for creating and managing ApeCloud DTS tasks. It can generate task configs, start dt-main, list tasks, stream logs, and stop, restart, or delete local task records.

For installation and detailed usage, see dt-cli/README.md.

Tutorial

• prerequisites
• mysql -> mysql
• pg -> pg
• mongo -> mongo
• redis -> redis
• mysql -> starrocks
• mysql -> doris
• mysql -> clickhouse
• mysql -> tidb
• mysql -> kafka -> consumer
• pg -> starrocks
• pg -> doris
• pg -> clickhouse
• pg -> kafka -> consumer
• snapshot + cdc without data loss
• modify data by lua

Run tests

Refer to test docs for details.

More docs

• Configurations
  • config details
• Structure tasks
  • migration
  • check
  • check by Liquibase
• Snapshot tasks
  • data migration
  • data check
  • data revise
  • data review
  • resume at breakpoint
  • multiple tables in parallel
• CDC tasks
  • data sync
  • heartbeat to source database
  • two-way data sync
  • generate sqls from CDC
  • resume at breakpoint
• Custom consumers
  • mysql/pg -> kafka -> consumer
• Data processing
  • modify data by lua
• Monitor
  • monitor info
  • position info
• Task templates
  • mysql -> mysql
  • pg -> pg
  • mongo -> mongo
  • redis -> redis
  • mysql/pg -> kafka
  • mysql -> starrocks
  • mysql -> doris
  • mysql -> clickhouse
  • pg -> starrocks
  • pg -> doris
  • pg -> clickhouse

Benchmark

• MySQL -> MySQL, Snapshot

Method	Node Specs	RPS(rows per second)	Source MySQL Load (CPU/Memory)	Target MySQL Load (CPU/Memory)
ape_dts	1c2g	71428	8.2% / 5.2%	211% / 5.1%
ape_dts	2c4g	99403	14.0% / 5.2%	359% / 5.1%
ape_dts	4c8g	126582	13.8% / 5.2%	552% / 5.1%
debezium	4c8g	4051	21.5% / 5.2%	51.2% / 5.1%

• MySQL -> MySQL, CDC

Method	Node Specs	RPS(rows per second)	Source MySQL Load (CPU/Memory)	Target MySQL Load (CPU/Memory)
ape_dts	1c2g	15002	18.8% / 5.2%	467% / 6.5%
ape_dts	2c4g	24692	18.1% / 5.2%	687% / 6.5%
ape_dts	4c8g	26287	18.2% / 5.2%	685% / 6.5%
debezium	4c8g	2951	20.4% / 5.2%	98% / 6.5%

• Image size

ape_dts:2.0.25-alpha.1	debezium/connect:2.7
71.4 MB	1.38 GB

• more benchmark details

Contributing

Structure

Modules

• dt-main: program entry

• dt-precheck: pre-check, to minimize interruptions during subsequent data operations by identifying issues early for fast failure

• dt-connector: extractors + sinkers for databases

• dt-pipeline: pipeline to connect extractors and sinkers

• dt-parallelizer: parallel algorithms

• dt-task: create extractors + sinkers + pipelines + parallelizers according to configurations

• dt-common: common utils, basic data structures, metadata management

• dt-tests: integration tests

• related sub module: mysql binlog connector in rust

Build

• Minimum supported Rust version (MSRV) The current minimum supported Rust version (MSRV) is 1.85.0.
• cargo build
• build images

Checklist

• run cargo clippy --all-targets --all-features --workspace fix all clippy issues.

Community

If you have any questions, you can reach out to us through:

• ApeDTS GitHub Discussions

• ApeDTS Wechat Account with note ape-dts: