history export: one record takes ~26s over a remote builder (223 sequential Content/Read RPCs) #4136

Description

buildx history export reads blobs from the builder's content store in many small sequential Content/Read RPCs. Over a remote builder each read is a network round trip, so exporting one build record takes tens of seconds. docker/build-push-action runs this export after every build to produce the job summary, so every CI build on a remote builder pays that cost.

For context, in our CI, we use a remote buildkit builder, to get consistent high cache hit rates and to not be limited by the compute of GH hosted runners.

Expected behaviour

Exporting one record reads each blob in a handful of RPCs, so the time is dominated by transfer, not by the number of round trips.

Actual behaviour

One record took 28.1s to export. 27s of that was 223 sequential Content/Read RPCs of 80–290 ms each:

The traces come from a build of master with a root span added to history export (the BuildKit client then traces its gRPC calls) plus a few phase spans.

Where the small reads come from

Each ReadAt on containerd's proxy content store opens its own Content/Read stream:

1. sanitizeTrace runs a json.Decoder directly over an io.SectionReader of the remote blob. Every decoder refill (sized to the decoder's buffer, so between ~512 bytes and the largest span) becomes one RPC:

   buildx/util/desktop/bundle/trace.go

   Lines 40 to 48 in c7748d0

   	func sanitizeTrace(ctx context.Context, mp *contentutil.MultiProvider, desc ocispecs.Descriptor) (*ocispecs.Descriptor, error) {
   	ra, err := mp.ReaderAt(ctx, desc)
   	if err != nil {
   	return nil, err
   	}
   	defer ra.Close()
   	buf := &bytes.Buffer{}
   	dec := json.NewDecoder(io.NewSectionReader(ra, 0, ra.Size()))

2. bundle.Export passes the content store to containerd's archive exporter, which copies each blob with io.Copy over content.NewReader, so it reads 32 KiB per RPC:

   buildx/util/desktop/bundle/export.go

   Line 62 in c7748d0

   if err := imgarchive.Export(ctx, mp, gz, imgarchive.WithManifest(desc), imgarchive.WithSkipDockerManifest()); err != nil {

   buildx/vendor/github.com/containerd/containerd/v2/core/images/archive/exporter.go

   Lines 475 to 484 in c7748d0

   	r, err := cs.ReaderAt(ctx, desc)
   	if err != nil {
   	return 0, fmt.Errorf("failed to get reader: %w", err)
   	}
   	defer r.Close()
   	// Verify digest
   	dgstr := desc.Digest.Algorithm().Digester()
   	n, err := io.Copy(io.MultiWriter(w, dgstr.Hash()), content.NewReader(r))

The cost is the number of reads times the round-trip time, so it doesn't show up against a local builder.

Proposed fix

Buffer both paths so each RPC fetches up to 4 MiB:

• sanitizeTrace decodes through bufio.NewReaderSize(io.NewSectionReader(ra, 0, ra.Size()), min(ra.Size(), 4 MiB)).
• bundle.Export wraps the provider it hands to imgarchive.Export so each ReaderAt it returns also has a Reader() backed by the same buffered reader. content.NewReader uses Reader() when it exists, so the exporter's io.Copy reads in 4 MiB chunks.

Memory stays bounded by one 4 MiB buffer, capped at the blob size. With this change, the same record:

Buildx version

github.com/docker/buildx c7748d0 (master, built from source). Also seen with v0.38.0 in the docker/build-push-action post step, where exporting the same record took 13.1s from a GitHub-hosted runner.

Builders list

NAME/NODE        DRIVER/ENDPOINT             STATUS    BUILDKIT   PLATFORMS
bench            remote
 \_ bench0        \_ tcp://127.0.0.1:17234   running   v0.28.0    linux/amd64 (+4), linux/386

The endpoint is the rootless buildkitd's unix socket, SSH-forwarded from a Kubernetes pod (~80–120 ms per RPC).

Configuration

$ docker buildx --builder bench history export <ref> -o record.dockerbuild

Any build record works; the more trace spans and log output, the more round trips.

Additional info

To capture the trace: add tracing.TraceCurrentCommand to the history export command (as build and bake already do), run it with OTEL_EXPORTER_OTLP_ENDPOINT pointing at Jaeger, and every Content/Read shows up as a span.

Assisted by Claude Code (Claude Opus 5.5). Reviewed and validated fully by myself.