README.md

English | 中文

Introduction

• ape-dts is a data migration tool enabling any-to-any data transfers.
• It also provides data subscription and data processing.
• It is lightweight, efficient and standalone, requiring no third-party components or extra storage.
• In Rust.

Key features

• Supports data migration between various databases, both homogeneous and heterogeneous.
• Supports snapshot and cdc tasks with resume from breakpoint.
• Supports checking and revising data.
• Supports filtering and routing at the database, table, and column levels.
• Implements different parallel algorithms for different sources, targets, and task types to improve performance.
• Allows loading user-defined Lua scripts to modify the data.
• Supports starting ape-dts as an HTTP server to pull data from the source, allowing users to use any language's HTTP client to retrieve and consume the data.

Supported task types

	mysql -> mysql	pg -> pg	mongo -> mongo	redis -> redis	mysql -> kafka	pg -> kafka	mysql -> starrocks	mysql -> clickhouse	mysql -> tidb	pg -> starrocks
Snapshot	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔
CDC	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔
Data check/revise/review	✔	✔	✔						✔
Structure migration	✔	✔					✔	✔	✔	✔

Quick starts

Tutorial

• prerequisites
• mysql -> mysql
• pg -> pg
• mongo -> mongo
• redis -> redis
• mysql -> starrocks
• mysql -> clickhouse
• mysql -> tidb
• mysql -> kafka -> consumer
• pg -> starrocks
• pg -> kafka -> consumer
• mysql -> ape_dts(HTTP server) -> consumer
• pg -> ape_dts(HTTP server) -> consumer
• snapshot + cdc without data loss
• modify data by lua

Run tests

Refer to test docs for details.

More docs

• Configurations
  • config details
• Structure tasks
  • migration
  • check
  • check by Liquibase
• Snapshot tasks
  • data migration
  • data check
  • data revise
  • data review
  • resume at breakpoint
• CDC tasks
  • data sync
  • heartbeat to source database
  • two-way data sync
  • generate sqls from CDC
  • resume at breakpoint
• Custom consumers
  • mysql/pg -> kafka -> consumer
  • mysql/pg -> ape_dts(HTTP server) -> consumer
• Data processing
  • modify data by lua
• Monitor
  • monitor info
  • position info
• Task templates
  • mysql -> mysql
  • pg -> pg
  • mongo -> mongo
  • redis -> redis
  • mysql/pg -> kafka
  • mysql/pg -> ape_dts(HTTP server)
  • mysql -> starrocks
  • mysql -> clickhouse
  • pg -> starrocks

Benchmark

• MySQL -> MySQL, Snapshot

Method	Node Specs	RPS(rows per second)	Source MySQL Load (CPU/Memory)	Target MySQL Load (CPU/Memory)
ape_dts	1c2g	71428	8.2% / 5.2%	211% / 5.1%
ape_dts	2c4g	99403	14.0% / 5.2%	359% / 5.1%
ape_dts	4c8g	126582	13.8% / 5.2%	552% / 5.1%
debezium	4c8g	4051	21.5% / 5.2%	51.2% / 5.1%

• MySQL -> MySQL, CDC

Method	Node Specs	RPS(rows per second)	Source MySQL Load (CPU/Memory)	Target MySQL Load (CPU/Memory)
ape_dts	1c2g	15002	18.8% / 5.2%	467% / 6.5%
ape_dts	2c4g	24692	18.1% / 5.2%	687% / 6.5%
ape_dts	4c8g	26287	18.2% / 5.2%	685% / 6.5%
debezium	4c8g	2951	20.4% / 5.2%	98% / 6.5%

• Image size

ape_dts:2.0.3	debezium/connect:2.7
86.4 MB	1.38 GB

• more benchmark details

Contributing

Structure

Modules

• dt-main: program entry

• dt-connector: extractors + sinkers for databases

• dt-pipeline: pipeline to connect extractors and sinkers

• dt-parallelizer: parallel algorithms

• dt-task: create extractors + sinkers + pipelines + parallelizers according to configurations

• dt-common: common utils, basic data structures, metadata management

• dt-tests: integration tests

• related sub module: mysql binlog connector in rust

Build

• cargo build
• build images

Checklist

• run cargo clippy --all-targets --all-features --workspace fix all clippy issues.

Contact us

Slack Community

Wechat Group