A modern, modular Python tool for cross-cloud storage backup and sync using rclone. This tool provides a clean, extensible architecture for managing backups across multiple cloud storage providers.
- Modular Architecture: Clean separation of concerns with dedicated modules
- Multi-Cloud Support: Works with AWS S3, Alibaba Cloud OSS, Tencent COS, Cloudflare R2, and more
- JSON Configuration: Simple, readable configuration format
- Real-time Progress: Monitor backup progress with detailed statistics
- Enhanced Connection Testing: Comprehensive diagnostics with bucket configuration verification
- Smart Error Handling: Detailed troubleshooting suggestions for connection failures
- Task Management: Organize backups into reusable tasks
- CLI Interface: Professional command-line interface with comprehensive options
- Zero Dependencies: Pure Python standard library implementation
- Type Safety: Full type hints and validation
- Error Handling: Comprehensive exception handling with custom error types
git clone <repository-url>
cd easy-rclone
pip install -e .pip install -e ".[dev]"- Python 3.8+
- rclone (install from https://rclone.org/downloads/)
easy-rclone check-rcloneeasy-rclone create-configEdit cloud_config.json with your cloud storage credentials:
{
"cloud_storages": [
{
"name": "tencent-cos",
"type": "s3",
"provider": "tencent",
"access_key_id": "your-secret-id",
"secret_access_key": "your-secret-key",
"region": "ap-shanghai",
"endpoint": "https://cos.ap-shanghai.myqcloud.com",
"bucket": "your-bucket"
},
{
"name": "cloudflare-r2",
"type": "s3",
"provider": "cloudflare",
"access_key_id": "your-access-key-id",
"secret_access_key": "your-secret-access-key",
"region": "auto",
"endpoint": "https://your-account-id.r2.cloudflarestorage.com",
"bucket": "your-bucket"
}
],
"backup_tasks": [
{
"name": "cos-to-r2",
"source": "tencent-cos:your-bucket/files/",
"destination": "cloudflare-r2:your-bucket/backup/",
"options": {
"sync": "true",
"transfers": "4",
"bwlimit": "10M"
}
}
]
}# Test all backup tasks
easy-rclone test-all
# Test specific storage connection with detailed diagnostics
easy-rclone test-connection tencent-cos
easy-rclone test-connection cloudflare-r2The test-connection command provides comprehensive diagnostics:
$ easy-rclone test-connection demo-cos
=== 测试存储连接: demo-cos ===
存储类型: s3
提供商: tencent
区域: ap-guangzhou
端点: https://cos.ap-guangzhou.myqcloud.com
存储桶: example-bucket-1234567890
测试连接路径: demo-cos:example-bucket-1234567890/
正在连接...
✓ 连接成功 - 找到 3 个文件/目录，大小: Total objects: 5
=== 详细信息 ===
文件数量: 3
路径大小: Total objects: 5
RClone版本: rclone v1.71.1
文件列表:
backup/
docs/
images/
=== Bucket配置检查 ===
✓ 存储桶配置正确 - 可以成功访问
✓ Bucket级别访问正常 - 找到 3 个项目easy-rclone run-task cos-to-r2easy-rclone check-rclone- Check if rclone is installedeasy-rclone create-config- Create sample configuration fileeasy-rclone list-storages- List all configured cloud storageseasy-rclone list-tasks- List all configured backup tasks
easy-rclone run-task <task-name>- Run specific backup taskeasy-rclone test-task <task-name>- Test specific backup taskeasy-rclone run-all- Run all backup taskseasy-rclone test-all- Test all backup tasks
easy-rclone test-connection <storage-name>- Test specific storage connection with detailed diagnostics- Shows storage configuration details
- Displays connection status and file statistics
- Verifies bucket configuration correctness
- Provides troubleshooting suggestions for connection failures
--config, -c <file>- Configuration file path (default: cloud_config.json)--log-level <level>- Log level: DEBUG, INFO, WARNING, ERROR (default: INFO)--no-progress- Disable progress display
rclone_backup/
├── __init__.py # Package initialization
├── config/ # Configuration management
│ ├── __init__.py
│ ├── manager.py # ConfigManager class
│ └── models.py # Data models (CloudStorage, BackupTask)
├── core/ # Core functionality
│ ├── __init__.py
│ ├── backup.py # BackupManager class
│ ├── storage.py # StorageManager class
│ └── rclone.py # RCloneExecutor class
├── utils/ # Utility modules
│ ├── __init__.py
│ ├── logger.py # Logging utilities
│ └── exceptions.py # Custom exceptions
└── cli/ # Command line interface
├── __init__.py
└── main.py # CLI entry point
- Handles JSON configuration loading and validation
- Provides configuration models with type safety
- Manages configuration file creation and updates
- Manages cloud storage configurations
- Generates rclone configuration files
- Handles storage-specific provider mappings
- Manages backup tasks and execution
- Provides task validation and testing
- Handles backup strategies (sync vs copy)
- Wraps rclone command execution
- Provides real-time progress monitoring
- Handles error processing and reporting
- s3: AWS S3, Alibaba Cloud OSS, Tencent COS, Cloudflare R2, etc.
- oss: Alibaba Cloud OSS (legacy)
The tool automatically maps provider names to rclone-compatible values:
tencent→TencentCOScloudflare→Cloudflarealiyun→Alibabahuawei→HuaweiOBSaws→AWS
sync: Use sync mode instead of copy (true/false)delete-mode: Delete files in destination that don't exist in source (true/false)transfers: Number of parallel transfers (1-64)checkers: Number of parallel checkers (1-64)bwlimit: Bandwidth limit (e.g., "10M" for 10MB/s)
from rclone_backup import ConfigManager, StorageManager, BackupManager, RCloneExecutor
# Initialize components
config_manager = ConfigManager("cloud_config.json")
config = config_manager.load_config()
rclone_executor = RCloneExecutor()
storage_manager = StorageManager()
storage_manager.load_from_config(config_manager)
backup_manager = BackupManager(rclone_executor, storage_manager)
backup_manager.load_from_config(config_manager)
# Create rclone config
storage_manager.create_rclone_config()
# Run backup task
success, message = backup_manager.run_task("cos-to-r2")
print(f"Backup result: {message}")
# Test connection
success, files = rclone_executor.list_remote("tencent-cos:your-bucket/")
if success:
print(f"Found {len(files)} files")Run the test suite:
# Run all tests
python -m pytest
# Run with coverage
python -m pytest --cov=rclone_backup
# Run specific test module
python -m pytest tests/test_config.py# Clone repository
git clone <repository-url>
cd easy-rclone
# Create virtual environment
python -m venv venv
source venv/bin/activate # Linux/macOS
# or
venv\Scripts\activate # Windows
# Install development dependencies
pip install -e ".[dev]"# Format code
black rclone_backup/
# Lint code
flake8 rclone_backup/
# Type checking
mypy rclone_backup/-
rclone not found
easy-rclone check-rclone # Install rclone from https://rclone.org/downloads/ -
Configuration errors
easy-rclone test-all # Check configuration syntax and values -
Connection failures
easy-rclone test-connection <storage-name> # Verify credentials and network connectivity
-
Permission denied
- Check API keys have correct permissions
- Verify bucket access policies
- Ensure network connectivity to cloud providers
- Fork the repository
- Create a feature branch:
git checkout -b feature-name - Make your changes and add tests
- Run the test suite:
python -m pytest - Commit your changes:
git commit -am 'Add new feature' - Push to the branch:
git push origin feature-name - Submit a pull request
This project is licensed under the MIT License - see the LICENSE file for details.
- rclone - The amazing command-line program that powers this tool
- All cloud storage providers for their excellent APIs and services
- The Python community for the fantastic standard library
Happy backing up! 🚀