ZSTD BACKUP

Backup your files and folders using ZSTD compression algorithm

Why Zstandard (ZSTD)

Zstandard (ZSTD) provides the best compression ratio (as far as I know).
Furthermore, the algorithm also allows multithreading for faster compression while being easy to use and is compatible with tar command

For more details, visit facebook/zstd

---

Table of contents

• Requirements
  1. Python
  2. Git
• Setup
  1. Clone repo
  2. Create virtual environment
  3. Install dependencies
  4. Edit configuration
  5. Execute program
• config.yml
• Issues
• License

---

Installation

Requirements

1. Python (3.8 - 3.13)
   • Linux: Included in most modern distros
   • Windows
   • MacOS
2. Git
   • Debian-based distros (Debian, Ubuntu, Mint, etc.): apt install git
   • RedHat-based distros (Red Hat, Fedora, CentOS, etc.): dnf install git
   • Arch-based distros (Arch Linux, Manjaro): pacman -S git
   • Git for Windows
   • Git for macOS

Setup

1. Clone this repository

   git clone --depth 1 https://github.com/knighthat/zstd-backup && cd zstd-backup

2. Create virtual environment (recommended, or use global python)

   • Windows

     Run this in your command prompt or powershell

     python -m venv venv

   • Linux

     Some distros require you to install venv module before executing this command

     /usr/bin/python3 -m venv venv

   • MacOS

     python -m venv venv

3. Install required packages

   venv/bin/pip install -r requirements.txt

4. Edit configuration config.yml

5. Run program by executing main.py script

   • Windows

     venv\Scripts\python zstd_backup.py

   • Others

     venv/bin/python3 main.py

config.yml

This is where you specify the files/folders that are included in the compressed file.
Also the place where compressed file saved to.

# Available levels: DEBUG, INFO, WARN, ERROR, FATAL
console_log_level: INFO

# All files and folders will be included
# in the final compressed file.
include:
  - '/path/to/compress'

# Where to save this backup.
# Must be a path (program creates dirs if path not exist)
destination: "./backups"

# Ignore all files that start with 1 of the paths below.
# Must be in absolute form.
# If '/path/to/ignore' is listed, any sub-folders and files
# within that directory will be ignored.
# Since v0.0.3, you can use regular expression to ignore files.
# For regex to match the entire path, use colons:
# - ':ignore-\d+:'
# 'ignore-1' matches the pattern but not '/path/to/ignore-1'
# For regex to find matches in a string, use semi-colon:
# - ';ignore-\d+;'
# will match both 'ignore-1' and '/path/to/ignore-1'
ignore: []

old_backups:
  # How many backups should I keep (default: 5, 0 means unlimited)
  keep: 5
  # If a backup's creation date exceeds this number, it'll be deleted.
  # Set this to 0 to skip this process
  retention: 30   # Days
  # Should this program delete older backups (even if it was
  # within retention days and total files less than 'keep')
  # to save space for new backup.
  # This option always keeps the most recent backup
  # unless 'aggressive' is set to True.
  remove_old_backups_for_space: true
  # If True, delete to the last backup to make some space for new backup
  aggressive: false

arguments:
  # Compression level.
  # Higher values are slower but yield smaller size.
  # Default level is 3
  # More at: https://python-zstandard.readthedocs.io/en/latest/compressor.html#zstdcompressor
  level: 3
  # How many cores should the algo use.
  # More threads equals faster compression time.
  # 0 will use all threads (also the default)
  threads: 4

settings:
  # Program will attempt to write to compressed
  # file this many bytes per cycle.
  # Higher number can cause I/O bottleneck
  # If you're unsure, leave it at default 1024
  write_chunk: 1024   # bytes
  # Settings related to progress bar
  # This function is still in Beta
  progress_bar:
    # Enabling progress bar may
    # result in slower write.
    enabled: true

# Send compressed file to remote server
remote_storage:
  enabled: false
  # As of version 0.1.0, only SFTP is supported
  type: SFTP
  server:
    host: '127.0.0.1'
    port: 22
  credentials:
    # Available protocols: PASSWORD, DSS, ED25519, RSA, ECDSA
    # PASSWORD requires presence of 'password' field,
    # others require 'keypath' and 'passphrase'
    # Leave 'passphrase' empty if key isn't encrypted with one.
    protocol: 'ED25519'
    # Automatically switch to public/private key
    # once 'private_key' is filled out.
    username: username
    keypath: ''
    passphrase: ''
  # Path must contain name you want the file to be
  # '{}' will be replaced by the original name.
  remote_path: '/remote/location/compressed.zstd'
  # File won't be deleted if error occurs during transfer
  delete_after_transfer: false

Issues

If you have any question, please open a ticket HERE

License

This repo is licensed under MIT. Feel free to use, modify, and re-distribute without restrictions.