We've now merged #1770 to add support for integration tests 🎉 However, buildx has been around for several years before these tests, so we don't have anywhere near the amount of test coverage that we want. These tests are essential to track regressions, detect changes in behavior as we refactor and build new functionality, and add new modes and drivers.

In general, all new functionality should be added with tests wherever feasible, but we need a plan to work through the existing backlog of code with no tests. This issue is to track what broad groups of tests we need, and how far along we are. This list isn't currently complete -- but as a general rule of thumb, if we document the behavior on https://docs.docker.com, we should be testing it.

Ideally, each sub-point in the list would end up as a PR, though maybe it makes sense to group some of the smaller ones together, or split some of the larger ones up.

(note to maintainers, feel free to add anything that you think is missing 🥳)

The first type of tests that we need are for the top-level commands package - at the time of writing, these currently reside in the top-level tests package. These are designed to test each of the buildx subcommands, such as build, bake, inspect, etc.

The areas that we need coverage for:

• Building with buildx build / buildx bake
  • From stdin tests: build from stdin #2377
  • Multi-platform builds (--platform) tests: build multiplatform #2620
  • Cache (--cache-to/--cache-from/--no-cache/--no-cache-filter)
  • Exporters (--output/--load/--push/--tag) build: handle push/load shorthands for multi exporters #2302
  • Build arguments / build contexts (--build-arg/--build-context) build: validate build args and labels #2040
  • Metadata recording (--metadata-file/--iidfile)
  • Cross node builds (builds that occur across multiple nodes in a builder)
  • Buildkit subrequests (--print) test: build print #2423
  • Secrets (--secret/--ssh) tests: build secret #2381
  • Shm size (--shm-size) test: build shm-size and ulimit #2279
  • Ulimit (--ulimit) test: build shm-size and ulimit #2279
  • Labels (--label) build: validate build args and labels #2040
  • Attestations (--attest/--provenance/--sbom) build: error on attests on non-multiplatform driver #1988
  • Progress output (--progress) tests: add basic build progress test #1925
    • Not sure quite what this would look like, but some sanity checks on stdout to ensure that progress is continually printed, the ticker keeps updating, any relevant log messages correctly appear, warnings are displayed at the end of build, etc.
  • Building using the controller interface (BUILDX_EXPERIMENTAL env var, --detach flag) tests: test with buildx experimental #2376
  • Tracing support
  • History API
  • Source policy
• Registry manipulation with buildx imagetools
  • Inspecting images (buildx imagetools inspect) imagetools: Allow annotations for OCI image index #1965 tests: add integration test for imagetools create #1978
  • Creating images (buildx imagetools create) tests: add imagetools tests for copying manifests and indexes #2013
• Debugging with the --invoke flag
  • Different values for the --invoke flag (debug-shell, on-error, shell commands)
  • Monitor mode commands (attach/list/help/etc)
  • Breakpoints, etc (when added)
• Disk management utilities with buildx du / buildx prune
• Versioning with buildx version (Integration test for docker buildx version #1985)
• Builder management with buildx create/buildx rm/buildx ls/buildx stop/buildx use
  • This is going to need a more manual driver creation process, that's less integrated into the sandbox probably.

The second type of tests that we need are for the top-level driver package. These drivers allow connecting to buildkit running in different configurations. Each driver supports lots of different types of options, and so we should have tests that test for configuring and setting up buildkit with these different options. Currently, we don't have any tests of this format.

The areas that we need coverage for:

• docker driver
  • overlayfs graph driver
  • containerd snapshotter (test: register docker worker with containerd snapshotter #1897)
• docker-container driver (see docs)
• kubernetes driver (see docs)
• remote driver (see docs)