Yandex Cloud S3 Multipart Upload Script

A robust bash script for uploading large files to Yandex Cloud Object Storage using the S3-compatible multipart API.

This script is designed to handle the upload of large files by splitting them into smaller, manageable parts. It uploads these parts individually and then reassembles them in the cloud, providing a more reliable and efficient way to transfer large datasets.

Features

• Multipart Upload: Reliably uploads files of any size, overcoming the typical 5GB single-file upload limit.
• Configurable Part Size: Easily configure the size of each part to optimize for your network conditions.
• Robust Error Handling: The script detects failures during part uploads and will safely abort the entire process to prevent incomplete or corrupt uploads.
• Automatic Cleanup: A temporary directory used for storing file parts is automatically created and cleaned up after the script finishes or fails.
• Dependency Check: Verifies that all required command-line tools are installed before execution.
• User-Friendly Output: Provides clear, step-by-step progress and error messages.

Prerequisites

Before using this script, you must have the following command-line tools installed and configured:

1. Yandex Cloud CLI (yc): The official command-line interface for Yandex Cloud.

   • Installation Guide: Yandex Cloud CLI Documentation
   • Ensure it is configured with a profile that has storage.editor or higher permissions for the target bucket. You can configure it by running yc init.

2. jq: A lightweight and flexible command-line JSON processor.

   • Installation:

     # On Debian/Ubuntu
     sudo apt-get update && sudo apt-get install jq
     
     # On macOS (using Homebrew)
     brew install jq
     
     # On RHEL/CentOS
     sudo yum install jq

3. split: A standard Unix utility for splitting files. It is typically pre-installed on most Linux and macOS systems.

Installation

You can use this script by either cloning the repository or downloading the script file directly.

Option 1: Clone the repository

git clone https://github.com/vloldik/yc-s3-multipart-upload.git
cd yc-s3-multipart-upload

Option 2: Download the script

curl -O https://raw.githubusercontent.com/vloldik/yc-s3-multipart-upload/main/upload.sh
chmod +x upload.sh

The script requires two mandatory arguments and accepts one optional argument:

• <file_path>: The full path to the large file you want to upload.
• <bucket_name>: The name of the Yandex Cloud Object Storage bucket.
• [part_size_in_MB]: (Optional) The size of each part in Megabytes. Defaults to 100 MB if not specified.

Examples

Basic Usage (with default 100MB part size):

./upload.sh /path/to/my/large-archive.zip my-yc-bucket

Custom Part Size: To upload a file with a part size of 250MB:

./upload.sh /path/to/my/huge-dataset.tar.gz my-yc-bucket 250