English | 中文

Introduction

• ape-dts is a tool aimed for any-to-any data migration.
• It is lightweight and does not rely on third-party components or additional storage.
• In Rust.

Tasks supported

	mysql -> mysql	pg -> pg	mongo -> mongo	redis -> redis
Snapshot	✔	✔	✔	✔
CDC	✔	✔	✔	✔
Data check/revise/review	✔	✔	✔
Structure migration/check	✔	✔

Quick start

Run demo in docker

Task configurations are in ini format. For more details, refer to config details, snapshot migration and CDC data sync.

docker run -it \
--entrypoint sh \
-v [absolute-path]/task_config.ini:/task_config.ini \
apecloud/ape-dts:0.1.13.hotfix4 \
-c "/ape-dts /task_config.ini"

Run tests

Refer to docs for more details.

More docs

• Configurations
  • config details
• Snapshot tasks
  • migration
  • check
  • revise
  • review
• CDC tasks
  • data sync
  • heartbeat
  • two-way data sync
• Data processing
  • custom lua script

Contributions

Structure

Modules

• dt-main: program entry
• dt-connector: extractors + sinkers for databases
• dt-pipeline: pipeline to connect extractors and sinkers
• dt-parallelizer: various parallel algorithms
• dt-task: create extractors + sinkers + pipelines + parallelizers according to configurations
• dt-common: common utils, basic data structures, metadata management
• dt-tests: integration tests

Coding

cargo build
cargo clippy --workspace

Build docker image

• arm64

docker buildx build \
--platform linux/arm64 --tag ape-dts:0.1.0-test-arm64 \
--build-arg MODULE_NAME=dt-main --load . 

• amd64

docker buildx build \
--platform linux/amd64 --tag ape-dts:0.1.0-test-amd64 \
--build-arg MODULE_NAME=dt-main --load . 

Contact us

Slack Community