Database Migration Tool

A developer-first database migration engine for MySQL → PostgreSQL migrations

Overview

A robust, asynchronous database migration engine that automatically processes MySQL to PostgreSQL migrations through a job queue system. Submit database URLs via API, track migration progress, and monitor job status in real-time.

Key Features

🚀 Asynchronous Processing - BullMQ-powered job queue for non-blocking migrations
📊 Real-time Tracking - Monitor job progress and status via REST API
🔄 Retry Mechanism - Automatic retry with exponential backoff
🏗️ Schema Introspection - Complete MySQL schema analysis and JSON export
🔒 Type-safe - Full TypeScript implementation
🐳 Docker Ready - Redis setup included with docker-compose
📦 Minimal Dependencies - Lightweight, fast, and maintainable

Quick Start

Prerequisites

Bun (latest version)
Redis (v7+)
Node.js 18+ (if not using Bun)

Installation

# Clone the repository
git clone https://github.com/yourusername/migration-tool.git
cd migration-tool

# Install dependencies
bun install

# Copy environment variables
cp .env.example .env

# Start Redis (using Docker)
docker compose up -d redis

# Run the application
bun run dev

The server will start at http://localhost:3000

Configuration

Create a .env file in the root directory:

# Server Configuration
PORT=3000
NODE_ENV=development

# Redis Configuration
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=

API Documentation

Root Endpoint

GET /

Response:

{
  "name": "Migration Tool API",
  "version": "1.0.0",
  "endpoints": {
    "health": "/api/health",
    "jobs": "/api/jobs"
  }
}

Health Check

GET /api/health

Response:

{
  "status": "ok",
  "timestamp": "2026-02-05T10:30:00.000Z"
}

Create Migration Job

POST /api/jobs/migration
Content-Type: application/json

{
  "databaseUrl": "mysql://user:pass@localhost:3306/source_db"
}

Response:

{
  "message": "Migration job created successfully",
  "jobId": "abc123",
  "queue": "migration"
}

Get Job Status

GET /api/jobs/:jobId/status

Response:

{
  "jobId": "abc123",
  "status": "processing",
  "progress": 60,
  "name": "run-migration",
  "data": {
    "databaseUrl": "mysql://user:pass@localhost:3306/source_db"
  }
}

Get Job Details

GET /api/jobs/:jobId

Response:

{
  "jobId": "abc123",
  "name": "run-migration",
  "status": "completed",
  "progress": 100,
  "createdAt": "2026-02-05T10:30:00.000Z",
  "completedAt": "2026-02-05T10:32:00.000Z"
}

List All Jobs

GET /api/jobs

Response:

{
  "jobs": [
    {
      "jobId": "abc123",
      "name": "run-migration",
      "status": "completed",
      "progress": 100
    },
    {
      "jobId": "def456",
      "name": "run-migration",
      "status": "processing",
      "progress": 45
    }
  ],
  "total": 2
}

Get Queue Statistics

GET /api/jobs/stats

Response:

{
  "migration": {
    "name": "migration",
    "count": 5,
    "waiting": 2,
    "active": 1,
    "completed": 2,
    "failed": 0
  }
}

Architecture

System Overview

src/
├── config/              # Configuration (queue, redis, app settings)
├── types/               # TypeScript type definitions
├── models/              # Data models (Redis persistence)
├── controllers/         # HTTP request handlers
├── routes/              # API route definitions
├── services/            # Business logic orchestration
├── middlewares/         # Express middlewares
├── utils/               # Utility functions (database, logger, migration)
└── queues/              # Job queue system (BullMQ)
    ├── jobs/            # Job creation & definitions
    ├── processors/      # Business logic (core work)
    ├── workers/         # BullMQ worker instances
    └── events/          # Event handlers (monitoring)

For detailed architecture documentation, see ARCHITECTURE.md

Migration Flow

Below is the end-to-end flow when a migration job is submitted:

User Request                        Queue System                         Database
─────────────                       ────────────                         ────────

POST /api/jobs/migration
  { databaseUrl }
        │
        v
  JobController
  .createMigrationJob()
        │
        v
  QueueService
  .addMigrationJob()
        │
        ├──→ migrationQueue.add()  ──→  Redis (BullMQ)
        │                                    │
        └──→ JobModel.create()     ──→  Redis (job:{id})
        │                                    │
        v                                    v
  Return { jobId }                  Worker picks up job
                                    (concurrency: 2)
                                             │
                                             v
                                    processMigrationJob()
                                             │
                                             ├──→ listDatabaseTables() ──→ MySQL/PostgreSQL
                                             │    (information_schema)
                                             │
                                             ├──→ getTableRelationships()──→ MySQL/PostgreSQL
                                             │    (PKs, FKs, reverse refs)
                                             │
                                             └──→ Migration Steps:
                                                  1. Validating
                                                  2. Extracting
                                                  3. Transforming
                                                  4. Loading
                                                  5. Verifying
                                                  (progress: 0→100%)
                                             │
                                             v
                                    QueueEvents fires:
                                    - completed (on success)
                                    - failed (on error)
                                    - progress (during execution)
                                             │
                                             v
                                    JobModel.updateStatus()
                                    ──→ Redis: job:{id}

Step-by-step breakdown:

API Layer — Client sends POST /api/jobs/migration with a databaseUrl. The controller validates the request and passes it to the service layer.
Job Creation — QueueService adds the job to the BullMQ migration queue and persists job metadata to Redis via JobModel.
Worker Processing — The migration worker (concurrency: 2) picks up the job and delegates to processMigrationJob().
Schema Introspection — The processor connects to the source database, lists all tables, and analyzes relationships (primary keys, foreign keys, reverse references).
Migration Steps — The job executes through 5 stages (Validating → Extracting → Transforming → Loading → Verifying), updating progress at each step.
Completion — BullMQ QueueEvents fires a completed event, which updates the job status in Redis. Failed jobs retry with exponential backoff (3 attempts).

Clients can poll GET /api/jobs/:jobId/status at any time to check progress.

Development

Available Scripts

# Development with hot reload
bun run dev

# Production build
bun run start

# Run tests
bun test

Development Tools

Redis Commander - Web-based Redis management UI

# Start Redis with Commander
docker compose up -d

# Access at http://localhost:8081

Project Structure

migration-tool/
├── index.ts                   # Application entry point
├── src/
│   ├── app.ts                 # Express app setup
│   ├── config/                # Configuration files
│   ├── controllers/           # Request handlers
│   ├── models/                # Data models
│   ├── routes/                # API routes
│   ├── services/              # Business logic
│   ├── queues/                # BullMQ queues & workers
│   ├── middlewares/           # Express middlewares
│   ├── utils/                 # Utility functions
│   └── types/                 # TypeScript types
├── docker-compose.yml         # Docker services
├── .env.example               # Environment template
└── package.json               # Dependencies

Manual Testing

# 1. Start Redis (in a separate terminal)
docker compose up -d redis

# 2. Start the server
bun run dev

# 3. Create a migration job
curl -X POST http://localhost:3000/api/jobs/migration \
  -H "Content-Type: application/json" \
  -d '{
    "databaseUrl": "mysql://user:pass@localhost:3306/source_db"
  }'

# Response:
# {
#   "message": "Migration job created successfully",
#   "jobId": "abc123xyz",
#   "queue": "migration"
# }

# 4. Check job status
curl http://localhost:3000/api/jobs/abc123xyz/status

# 5. Get all jobs
curl http://localhost:3000/api/jobs

# 6. Get queue statistics
curl http://localhost:3000/api/jobs/stats

# 7. View Redis queue (using Redis CLI)
docker exec -it local_redis redis-cli
> KEYS bull:migration:*
> HGETALL bull:migration:1

Docker Deployment

Using Docker Compose

# Start all services (Redis + App)
docker compose up -d

# View logs
docker compose logs -f

# Stop services
docker compose down

Manual Docker Build

# Build image
docker build -t migration-tool .

# Run container
docker run -p 3000:3000 \
  -e REDIS_HOST=redis \
  -e REDIS_PORT=6379 \
  migration-tool

Roadmap

Current Features

✅ Asynchronous job queue system powered by BullMQ
✅ REST API for job management
✅ Real-time job status tracking and progress monitoring
✅ Schema introspection for MySQL databases
✅ Table relationship analysis (primary keys, foreign keys, reverse references)
✅ Automatic retry with exponential backoff (3 attempts)
✅ Redis-based job persistence and monitoring
✅ Support for hot-reloading in development (Bun)
✅ Docker support with Redis Commander UI
✅ TypeScript for type safety

Planned Features

PostgreSQL schema generation and target database creation
Data migration with batching and progress tracking
Migration validation and verification
Rollback capability for failed migrations
Web UI dashboard for job monitoring and management
Support for additional database types (SQL Server, Oracle, etc.)
Migration templates and presets for common scenarios
Webhook notifications for job completion/failure

Troubleshooting

Redis Connection Issues

# Check if Redis is running
docker ps | grep redis

# Test Redis connection
redis-cli ping

# Restart Redis
docker compose restart redis

# Check the app logs for Redis connection errors

Port Already in Use

# Change PORT in .env file
PORT=3001

# Or kill the process using the port
lsof -ti:3000 | xargs kill -9

Job Stuck in Processing

Jobs automatically retry with exponential backoff. To troubleshoot:

# View application logs
bun run dev

# Check Redis queue with Redis CLI
docker exec -it local_redis redis-cli

# List all migration queue keys
> KEYS bull:migration:*

# Inspect a specific job
> HGETALL bull:migration:{jobId}

# Check queue status
> LLEN bull:migration:waiting
> LLEN bull:migration:active

Common causes:

Check if Redis is running and accessible
Review application logs for error messages
Verify the database connection string
Look for jobs in the failed state using GET /api/jobs/stats
Inspect failed job logs using the job ID endpoint

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

Related Resources

Job Queue Details: See README.md in queues
Architecture Documentation: See ARCHITECTURE.md
Code Style Guide: See CLAUDE.md
Changelog: See CHANGELOG.md
Security Policy: See SECURITY.md

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

BullMQ - Robust job queue system
Express - Web framework
Bun - Fast JavaScript runtime
TypeScript - Type-safe JavaScript

Support & Community

🐛 Report Issues: GitHub Issues
💬 Discussions: GitHub Discussions
📚 Documentation: See ARCHITECTURE.md for detailed technical information

Built with ❤️ by Soumik Sarkar

Made with TypeScript • Powered by BullMQ • Running on Bun

Name		Name	Last commit message	Last commit date
Latest commit History 27 Commits
.cursor		.cursor
.github		.github
src		src
.env.example		.env.example
.gitignore		.gitignore
ARCHITECTURE.md		ARCHITECTURE.md
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
bun.lock		bun.lock
database_seed.js		database_seed.js
docker-compose.yml		docker-compose.yml
index.ts		index.ts
package.json		package.json
pnpm-lock.yaml		pnpm-lock.yaml
schema.sql		schema.sql
tsconfig.json		tsconfig.json

License

Soum-ik/database-migration-tool

Folders and files

Latest commit

History

Repository files navigation

Database Migration Tool

Overview

Key Features

Table of Contents

Quick Start

Prerequisites

Installation

Configuration

API Documentation

Root Endpoint

Health Check

Create Migration Job

Get Job Status

Get Job Details

List All Jobs

Get Queue Statistics

Architecture

System Overview

Migration Flow

Development

Available Scripts

Development Tools

Project Structure

Manual Testing

Docker Deployment

Using Docker Compose

Manual Docker Build

Roadmap

Current Features

Planned Features

Troubleshooting

Redis Connection Issues

Port Already in Use

Job Stuck in Processing

Contributing

Related Resources

License

Acknowledgments

Support & Community

About

Topics

Resources

License

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages