s3go

Inspired by s3p, s3go is a lightweight command-line tool for high-throughput inspection of S3-compatible buckets. It specialises in listing and comparing very large keyspaces while keeping memory use minimal.

Features

• Fast object listing – scans keys in parallel.
• Bucket comparison mode – optionally target a destination bucket to spot:
  • objects present in the source but missing in the destination
  • objects with differing ETags (multipart uploads with - in the ETag are ignored when sizes match)
  • counts of matched, mismatched, and missing objects
• Streaming CSV output – segments flush to disk immediately; no large in-memory buffers or resorting passes.
• Flexible endpoints and credentials – configure any S3-compatible deployment via environment variables.

Why It’s Fast

• Parallel bisection – the keyspace is split recursively so multiple workers issue ListObjectsV2 calls on disjoint ranges.
• Sustained concurrency – workers keep the S3 pipeline full, minimising idle time between requests.
• No aggregation overhead – results stream directly to CSV files, avoiding costly in-memory merges.
• Segment-aware comparison – destination lookups reuse the same ranged listing, limiting redundant fetches.

Benchmarks

Bucket with ~1.2M objects:

Operation	Tool / Threads	Time	Speedup
Listing	rclone	4m40s	–
	s3go (50)	28s	~10x
Compare	rclone	23m38s	–
	s3go (50)	80s	~18x

Quick Start

1. Copy .env.example to .env and populate source credentials and endpoint details.
   Set destination environment variables if you plan to compare buckets.
2. Build or run the CLI:

   go run ./cmd/s3go \
     --bucket my-source \
     --prefix optional/prefix \
     --output-dir ./result \
     --dest-bucket my-destination \
     --missing-output missing.csv \
     --mismatch-output etag_mismatch.csv

3. Inspect the generated files inside your chosen output directory.

Additional flags include --start-after, --workers, --page-size, and --log-level. Consult cmd/s3go/main.go for full flag descriptions.