go-prefetch

A fast Go module downloader for CI pipelines. Pre-populates your Go module cache before go build runs.

Experimental: This project was created for educational purposes to explore the Go build pipeline internals. Use at your own risk in production environments.

Use Case

This tool is designed for CI/CD environments where builds start with a cold module cache. It's less useful for local development where your cache persists between builds.

Ideal for:

• GitHub Actions, GitLab CI, Jenkins, etc.
• Docker builds without layer caching
• Fresh VM/container builds

Why Faster?

Go's go mod download has several bottlenecks that this tool avoids:

• Async I/O — Go uses synchronous I/O that doesn't scale with GOMAXPROCS (benchmarks show it plateaus). We use tokio's async runtime which efficiently handles 64+ concurrent downloads.
• No per-module locking — Go serializes download→hash→extract per module with locks. We process modules independently.
• Fewer zip passes — Go opens the zip 3 times (validate, hash, extract). We read once for hashing, once for extraction.
• Connection pooling — Reuses TCP/TLS connections to the proxy.
• Streaming pipeline — Downloads and extractions overlap. Go waits for all downloads before extracting.
• No GOSUMDB lookups — Go verifies unknown modules against the checksum database. We trust go.sum directly.

Benchmarks

go-ethereum (170 modules, 188 MB):

Concurrency	go mod download	go-prefetch	Speedup
8	5.63s	2.81s	2.00x
16	5.33s	3.02s	1.77x
32	4.90s	2.56s	1.91x
64	4.59s	2.35s	1.95x

kubernetes (201 modules, 110 MB):

Concurrency	go mod download	go-prefetch	Speedup
16	2.66s	2.51s	1.06x
32	2.97s	2.52s	1.18x
64	2.84s	2.31s	1.23x

Speedup varies by project. Larger modules and slower networks benefit more.

Installation

Download from GitHub Releases, or:

cargo install go-prefetch

Usage

go-prefetch            # prefetch modules for current directory
go build ./...         # build with warm cache

GitHub Actions

- name: Prefetch modules
  run: |
    curl -sL https://github.com/cfal/go-prefetch/releases/latest/download/go-prefetch-x86_64-unknown-linux-musl.tar.gz | tar xz
    ./go-prefetch

- name: Build
  run: go build ./...

Options

go-prefetch [OPTIONS] [PATH]

Arguments:
  [PATH]                  Path to Go project (default: .)

Options:
  -c, --concurrency <N>   Max concurrent downloads (default: 64)
  -v, --verbose           Show detailed progress
  -q, --quiet             Suppress output except errors
      --dry-run           Show what would be downloaded
      --proxy <URL>       Override GOPROXY
      --continue-on-error Continue even if some modules fail

How It Works

1. Parses go.sum for the dependency list with expected hashes
2. Filters out private modules (respects GOPRIVATE)
3. Checks GOMODCACHE to skip already-cached modules
4. Downloads and extracts using a streaming pipeline (bounded memory)
5. Verifies h1: hashes against go.sum

The cache is 100% compatible with Go's toolchain—builds produce byte-identical binaries.

Limitations

• Public modules only — private modules are skipped (use go mod download for those)
• Unix only — requires Unix file permissions
• No go.work — workspaces not supported
• No VCS fetching — skips direct in GOPROXY

If go-prefetch fails, go build will download modules normally.

License

MIT