msync - Fast File Synchronization Tool

A high-performance, cross-platform file synchronization tool similar to rsync, built in Go. msync provides fast, reliable file synchronization with multiple comparison methods, concurrent processing, and comprehensive options for backup and restore operations.

Features

Core Capabilities

Multiple Comparison Methods: Choose between modification time, checksum (SHA256), or file size
High Performance: Multi-threaded processing with configurable worker pools
Cross-Platform: Runs on Linux, macOS, Windows, and other Unix-like systems
Preserve Attributes: Maintains file modification times and permissions
Directory Synchronization: Full recursive directory tree synchronization
TAR Archive Support: Create, extract, and synchronize TAR archives with optional compression
GPG Integration: Encrypt and sign TAR archives with GPG for secure backups
Dry Run Mode: Preview operations before execution
Delete Support: Remove extraneous files from destination
Progress Reporting: Detailed statistics and throughput information
Verbose Output: Comprehensive logging of operations

Performance Features

Concurrent Processing: Configurable number of worker threads
Efficient Checksumming: SHA256 hashing for content verification
Memory Efficient: Optimized file operations and streaming
Fast File Walking: Efficient directory tree traversal

Installation

Prerequisites

Go 1.19+ for building from source
Task (optional, for development): Install with brew install go-task/tap/go-task or nix-env -iA nixpkgs.go-task
GPG (optional, for encrypted archives): Usually pre-installed on Unix systems

From Source

git clone https://github.com/osmontero/msync.git
cd msync
task build
task install

Using Go

# Install with proper binary name
go install github.com/osmontero/msync/cmd@latest
# The binary will be installed as 'cmd', rename it:
mv $GOPATH/bin/cmd $GOPATH/bin/msync
# Or if using Go modules (Go 1.16+):
mv $(go env GOPATH)/bin/cmd $(go env GOPATH)/bin/msync

# Alternative: Build locally and install
git clone https://github.com/osmontero/msync.git
cd msync && task build && task install

Usage

Basic Synchronization

# Sync source directory to destination
msync /path/to/source /path/to/destination

# Using flags
msync --source /path/to/source --dest /path/to/destination

Advanced Examples

Checksum-based Synchronization

# Use SHA256 checksums for comparison (more accurate)
msync --checksum /source /dest
msync --method checksum /source /dest

High-Performance Sync

# Use 16 worker threads for large datasets
msync --threads 16 /source /dest

Backup with Deletion

# Mirror source to destination, removing extra files
msync --delete /source /dest

Preview Mode (Plane Mode)

# Preview what would be synchronized (comprehensive summary)
msync --plan --verbose /source /dest

# Interactive mode with preview and confirmation
msync --interactive /source /dest

# Traditional dry run (still supported)
msync --dry-run --verbose /source /dest

Verbose Monitoring

# Show detailed progress and statistics
msync --verbose /source /dest

Command Line Options

Usage:
  msync [OPTIONS] SOURCE DEST
  msync -s SOURCE -d DEST [OPTIONS]

Options:
  -s, --source PATH       Source directory or file
  -d, --dest PATH         Destination directory or file
  -c, --checksum          Use checksum comparison (slower but more accurate)
  -n, --dry-run           Show what would be synced without making changes
      --plan              Preview changes without executing (enhanced dry-run)
  -i, --interactive       Show preview and ask for confirmation before proceeding
  -v, --verbose           Enable verbose output
  -r, --recursive         Sync directories recursively (default: true)
      --delete            Delete files in destination not present in source
  -j, --threads N         Number of concurrent threads (default: 4)
      --method METHOD     Comparison method: mtime, checksum, size (default: mtime)
      --skip-broken-links Skip broken symbolic links entirely

TAR Archive Support:
      --tar-compress      Use gzip compression for TAR files
      --gpg-encrypt       Encrypt TAR files with GPG
      --gpg-sign          Sign TAR files with GPG  
      --gpg-key ID        GPG key ID for encryption/signing
      --gpg-keyring PATH  Path to GPG keyring

General:
  -h, --help              Show help message
      --version           Show version information

Comparison Methods

Method	Description	Speed	Accuracy	Use Case
`mtime`	Modification time + size	Fast	Good	General sync, frequent updates
`checksum`	SHA256 hash	Slow	Excellent	Critical data, verification
`size`	File size only	Very Fast	Basic	Large files, quick checks

TAR Archive Workflows

msync automatically detects TAR files and handles three types of operations:

Source	Destination	Operation	Description
Directory	`.tar`	Create	Create TAR archive from directory
`.tar`	Directory	Extract	Extract TAR archive to directory
`.tar`	`.tar`	Sync	Extract source, sync, create destination

Supported Extensions: .tar, .tar.gz, .tgz, .tar.gpg, .tar.gz.gpg, .tgz.gpg

Examples

Regular Backup

# Daily backup with progress monitoring
msync -v /home/user/documents /backup/documents

# Mirror backup (exact copy)
msync --delete -v /home/user/projects /backup/projects

Content Verification

# Verify backup integrity using checksums
msync --method checksum --dry-run /source /backup

High-Performance Scenarios

# Large dataset synchronization
msync -j 16 -v /data/large_dataset /backup/large_dataset

# Network storage synchronization
msync --threads 8 --method size /local/data /network/storage/data

Development Workflows

# Sync build output to deployment directory
msync --verbose ./build/ /var/www/html/

# Preview deployment changes with enhanced summary
msync --plan --delete ./build/ /var/www/html/

# Interactive deployment with confirmation
msync --interactive --delete ./build/ /var/www/html/

TAR Archive Operations

# Create TAR archive from directory
msync /src backup.tar

# Create compressed TAR archive
msync --tar-compress /src backup.tar.gz

# Extract TAR archive to directory
msync archive.tar /dst

# Create encrypted TAR archive
msync --gpg-encrypt --gpg-key USER_ID /src backup.tar.gpg

# Create signed and compressed TAR archive
msync --gpg-sign --tar-compress --gpg-key USER_ID /src backup.tar.gz

# TAR to TAR synchronization
msync old.tar.gz new.tar.gz

# Secure backup workflow
msync --gpg-encrypt --gpg-sign --tar-compress --gpg-key USER_ID /important/data secure_backup.tar.gz.gpg

Enhanced Preview Features

Comprehensive Preview Summary

The --plan flag provides an enhanced dry-run experience with:

Visual Summary: Clear breakdown of planned operations
Data Transfer Estimates: Shows net data transfer and estimated time
Operation Categorization: Separate counts for files to copy, delete, and directories to create
Smart Analysis: Detects when no changes are needed

# Example output:
============================================================
                    SYNC PREVIEW SUMMARY
============================================================
📋 PLANNED OPERATIONS:
------------------------------
📁 Files to copy:      15 (2.3 MB)
📂 Directories to create: 3
🗑️  Files to delete:    2 (150 KB)
------------------------------
📊 SUMMARY:
   Total operations:   20
   Files checked:      156
   Net data transfer:  2.2 MB
   Analysis time:      0.2s
   Estimated sync time: 1.5s
============================================================

Interactive Mode

The --interactive flag combines preview with user confirmation:

Performs comprehensive analysis
Shows detailed preview summary
Prompts for user confirmation
Executes only if approved

msync --interactive /source /dest
# Shows preview, then asks: "Do you want to proceed? [y/N]"

Troubleshooting

Common Issues

Broken Symbolic Links

If you see errors like "Failed to calculate checksum for ... no such file or directory", these are typically broken symbolic links (common in Python virtual environments):

# Skip broken symlinks entirely
msync --skip-broken-links /source /dest

# Or use verbose mode to see warnings but continue processing
msync --verbose /source /dest

Python Virtual Environments

Python venv/virtualenv directories contain symlinks that may be broken when moved between systems:

# Skip broken symlinks when syncing Python projects
msync --skip-broken-links /python-project /backup/python-project

Permission Errors

Ensure you have read permissions on source files and write permissions on destination:

# Check permissions
ls -la /source
ls -la /destination

# Run with verbose mode to see detailed error information
msync --verbose /source /dest

Performance

Benchmarks

On a modern system with SSD storage:

Checksum calculation: ~745µs per MB
File map building: ~300µs for 100 files
Synchronization: ~1.2ms for 50 files

Optimization Tips

Use appropriate thread count: Set --threads based on your storage and CPU
Choose the right method: Use mtime for speed, checksum for accuracy
Batch operations: Sync larger datasets in single operations
Monitor throughput: Use --verbose to track performance
Handle broken symlinks: Use --skip-broken-links to skip broken symbolic links (common in Python venv directories)

Development

Building from Source

# Clone repository
git clone https://github.com/osmontero/msync.git
cd msync

# Install dependencies
task deps

# Build binary
task build

# Run tests
task test

# Run benchmarks
task bench

# Build for multiple platforms
task build-all

# TAR-specific tests
task test-tar

# Run TAR demos
task demo-tar-basic
task demo-tar-compressed

Testing

# Run all tests
task test

# Run unit tests for specific packages
go test ./pkg/sync
go test ./pkg/tar

# Run benchmarks
task bench

# Test coverage
task test-coverage

# TAR-specific testing
task test-tar

# GPG functionality demos (requires GPG setup)
task gpg-generate-key
task demo-tar-encrypted
task demo-tar-signed

Project Structure

msync/
├── cmd/                    # Main application
├── pkg/
│   ├── sync/              # Core synchronization library
│   └── tar/               # TAR archive handling with GPG support
├── internal/utils/        # Utility functions
├── Taskfile.yaml         # Task automation (replaces Makefile)
└── README.md             # This file

Contributing

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

Development Guidelines

Write tests for new features
Follow Go coding conventions
Update documentation for API changes
Run benchmarks for performance-critical changes

License

This project is licensed under the GNU General Public License v3.0 - see the LICENSE file for details.

Comparison with rsync

Feature	msync	rsync
Language	Go	C
Performance	High (multi-threaded)	High (single-threaded)
Cross-platform	Native binaries	Requires compilation
Checksums	SHA256	MD5/SHA1
Configuration	Command-line flags	Many options
Network sync	Local only	SSH/network support
Compression	No	Yes
Incremental	Yes	Yes

Recent Updates

✅ TAR Archive Support (v1.1.0)

TAR Operations: Create, extract, and synchronize TAR archives
Compression Support: Gzip compression (.tar.gz, .tgz)
GPG Integration: Encrypt and sign TAR archives for secure backups
Automatic Detection: Smart detection of TAR files by extension
Flexible Workflows: Directory ↔ TAR, TAR ↔ TAR synchronization

Supported TAR Formats

.tar - Basic TAR archive
.tar.gz / .tgz - Gzip compressed TAR
.tar.gpg - GPG encrypted TAR
.tar.gz.gpg / .tgz.gpg - Compressed and encrypted TAR

Roadmap

Support

Issues: GitHub Issues
Discussions: GitHub Discussions
Documentation: Wiki

msync - Making file synchronization fast, reliable, and simple.

Name		Name	Last commit message	Last commit date
Latest commit History 13 Commits
cmd		cmd
internal/utils		internal/utils
pkg		pkg
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
Taskfile.yaml		Taskfile.yaml
go.mod		go.mod

License

osmontero/msync

Folders and files

Latest commit

History

Repository files navigation

msync - Fast File Synchronization Tool

Features

Core Capabilities

Performance Features

Installation

Prerequisites

From Source

Using Go

Usage

Basic Synchronization

Advanced Examples

Checksum-based Synchronization

High-Performance Sync

Backup with Deletion

Preview Mode (Plane Mode)

Verbose Monitoring

Command Line Options

Comparison Methods

TAR Archive Workflows

Examples

Regular Backup

Content Verification

High-Performance Scenarios

Development Workflows

TAR Archive Operations

Enhanced Preview Features

Comprehensive Preview Summary

Interactive Mode

Troubleshooting

Common Issues

Broken Symbolic Links

Python Virtual Environments

Permission Errors

Performance

Benchmarks

Optimization Tips

Development

Building from Source

Testing

Project Structure

Contributing

Development Guidelines

License

Comparison with rsync

Recent Updates

✅ TAR Archive Support (v1.1.0)

Supported TAR Formats

Roadmap

Support

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 1

Uh oh!

Contributors 2

Uh oh!

Languages