Skip to content

Document custom S3 endpoint support via URL query parameters #80

@corylanou

Description

@corylanou

Documentation for Custom S3 Endpoint Support

Related PR: benbjohnson/litestream#731

Once the PR is merged, we need to document the new custom S3 endpoint support feature that allows users to specify S3-compatible storage endpoints directly from the command line using URL query parameters.

Documentation Updates Needed

1. Update the Replication Guide

Add a new section about custom S3-compatible storage providers showing how to use query parameters.

URL Format

s3://bucket/path?endpoint=<endpoint>&region=<region>&forcePathStyle=<true|false>&skipVerify=<true|false>

Query Parameters

Parameter Description Default
endpoint Custom S3-compatible endpoint URL -
region AWS region us-east-1 for custom endpoints
forcePathStyle Use path-style addressing true for custom endpoints
skipVerify Skip TLS certificate verification false

2. Add Provider Examples

MinIO

# Replicate to MinIO
litestream replicate /path/to/db.sqlite \
  "s3://mybucket/db?endpoint=localhost:9000&forcePathStyle=true"

# Restore from MinIO
litestream restore -o restored.db \
  "s3://mybucket/db?endpoint=localhost:9000"

Tigris (Fly.io)

# Replicate to Tigris
litestream replicate /path/to/db.sqlite \
  "s3://my-tigris-bucket/db?endpoint=fly.storage.tigris.dev&region=auto"

# Restore from Tigris
litestream restore -o restored.db \
  "s3://my-tigris-bucket/db?endpoint=fly.storage.tigris.dev&region=auto"

Wasabi

litestream replicate /path/to/db.sqlite \
  "s3://mybucket/db?endpoint=s3.wasabisys.com&region=us-east-1"

Self-Hosted with Self-Signed Certificate

litestream replicate /path/to/db.sqlite \
  "s3://mybucket/db?endpoint=storage.internal.company.com&skipVerify=true"

3. Update the Restore Guide

Add examples showing restore operations with custom endpoints (as shown above).

4. Configuration File Alternative

Document that users can still use the configuration file for more complex setups:

dbs:
  - path: /path/to/db.sqlite
    replicas:
      - type: s3
        bucket: mybucket
        path: db
        endpoint: localhost:9000
        region: us-east-1
        force-path-style: true
        skip-verify: true
        access-key-id: YOUR_ACCESS_KEY
        secret-access-key: YOUR_SECRET_KEY

5. Backward Compatibility Note

Emphasize that all existing S3 URLs continue to work without modification:

# Standard AWS S3 - works exactly as before
litestream replicate /path/to/db.sqlite s3://mybucket/db

# Or with explicit region
litestream replicate /path/to/db.sqlite s3://mybucket.s3.us-west-2.amazonaws.com/db

6. Supported Providers Section

Create a list of tested S3-compatible providers:

  • MinIO
  • Tigris (Fly.io)
  • Wasabi
  • Backblaze B2 (S3-compatible API)
  • DigitalOcean Spaces
  • Scaleway Object Storage
  • Ceph with S3 gateway
  • Any other S3-compatible storage service

Notes

  • When using a custom endpoint, forcePathStyle defaults to true as most S3-compatible services require path-style addressing
  • The region parameter may be required by some services even if they don't use AWS regions
  • Use skipVerify=true only for development or when you trust the self-signed certificate
  • If no scheme is provided in the endpoint, http:// is automatically prepended

This feature addresses the long-standing issue #219 and enables users to work with any S3-compatible storage provider directly from the command line without needing a configuration file.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions