Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bake: use shared session for local sources for multiple targets #2615

Merged
merged 1 commit into from
Aug 7, 2024

Conversation

tonistiigi
Copy link
Member

@tonistiigi tonistiigi commented Jul 19, 2024

fixes #1377
fixes #92
depends on #2607

Detect cases where multiple bake targets would use the same
local source. For such cases a separate session request is
made in addition to session per target and local source
is made available in that source as well.

The new sessionID is sent with the request so the frontend
can ask associate it with the local source it needs.

The sources are still available in the main request session
as well. This would be used if frontend ignores the local-sessionid
parameter and makes sure that old version continue working.

> docker buildx bake lint validate-golangci

Before:

 => [validate-golangci internal] load build definition from lint.Dockerfile                                                        0.0s
 => => transferring dockerfile: 2.36kB                                                                                             0.0s
...
 => [validate-golangci internal] load build definition from lint.Dockerfile                                                        0.1s
 => => transferring dockerfile: 2.36kB                                                                                             0.0s
 => [lint internal] load build definition from lint.Dockerfile                                                                     0.1s
 => => transferring dockerfile: 2.36kB                                                                                             0.0s
 => [lint internal] load metadata for docker.io/library/golang:1.22-alpine                                                         0.2s
 => [validate-golangci internal] load metadata for docker.io/tonistiigi/xx:1.3.0                                                   0.3s
 => [validate-golangci internal] load .dockerignore                                                                                0.0s
 => => transferring context: 45B                                                                                                   0.0s
 => [lint internal] load .dockerignore                                                                                             0.1s
 => => transferring context: 45B                                                                                                   0.0s
...
 => [validate-golangci internal] load build context                                                                                0.6s
 => => transferring context: 960.30kB                                                                                              0.5s
...
 => [lint internal] load build context                                                                                             3.9s
 => => transferring context: 281.76MB
 
After:

  => [lint internal] load build definition from lint.Dockerfile                                                                     0.0s
  => => transferring dockerfile: 2.36kB                                                                                             0.0s
  => [lint internal] load metadata for docker.io/library/golang:1.22-alpine                                                         0.4s
  => [lint internal] load metadata for docker.io/tonistiigi/xx:1.3.0                                                                0.4s
  => [lint internal] load .dockerignore                                                                                             0.0s
  => => transferring context: 45B                                                                                                   0.0s
  => [validate-golangci golang-base 1/2] FROM docker.io/library/golang:1.22-alpine@sha256:8c9183f715b0b4eca05b8b3dbf59766aaedb41ec  0.0s
  => => resolve docker.io/library/golang:1.22-alpine@sha256:8c9183f715b0b4eca05b8b3dbf59766aaedb41ec07477b132ee2891ac0110a07        0.0s
  => [validate-golangci internal] load build context                                                                                0.5s
  => => transferring context: 957.98kB                                                                                              0.4s
  => [validate-golangci xx 1/1] FROM docker.io/tonistiigi/xx:1.3.0@sha256:904fe94f236d36d65aeb5a2462f88f2c537b8360475f6342e7599194  0.0s
  => => resolve docker.io/tonistiigi/xx:1.3.0@sha256:904fe94f236d36d65aeb5a2462f88f2c537b8360475f6342e7599194f291fb7e               0.0s
  => CACHED [validate-golangci golang-base 2/2] RUN apk add --no-cache git gcc musl-dev                                             0.0s
  => CACHED [validate-golangci lint-base 1/3] RUN wget -O- -nv https://raw.githubusercontent.com/golangci/golangci-lint/master/ins  0.0s
  => CACHED [validate-golangci lint-base 3/3] WORKDIR /go/src/github.com/docker/buildx                                              0.0s
  => CACHED [lint lint 1/1] RUN --mount=target=/go/src/github.com/docker/buildx     --mount=target=/root/.cache,type=cache,id=lint  0.0s
  => CACHED [lint lint-base 2/3] COPY --link --from=xx / /                                                                          0.0s
  => CACHED [validate-golangci validate-golangci 1/1] RUN --mount=target=/go/src/github.com/docker/buildx   golangci-lint config v  0.0s

This requires BuildKit/Dockerfile frontend with moby/buildkit#5174 support to work.

Note that this only works if the targets are using the same files from local context. If they ask for a different set of files then still multiple transfers will happen(on the same session). Atm. Dockerfile will try to pull all the files it needs from build context with a single transfer because every extra transfer results in new disk scanning overhead from the client side.

@dgageot

@tonistiigi tonistiigi changed the title Bake shared transfer sessions bake: use shared session for local sources for multiple targets Jul 19, 2024
Detect cases where multiple bake targets would use the same
local source. For such cases a separate session request is
made in addition to session per target and local source
is made available in that source as well.

The new sessionID is sent with the request so the frontend
can ask associate it with the local source it needs.

The sources are still available in the main request session
as well. This would be used if frontend ignores the local-sessionid
parameter and makes sure that old version continue working.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
@tonistiigi tonistiigi force-pushed the bake-shared-transfer-sessions branch from 3390d92 to b62d64b Compare July 24, 2024 17:58
@tonistiigi tonistiigi marked this pull request as ready for review July 24, 2024 17:58
@crazy-max crazy-max self-requested a review July 25, 2024 16:24
@crazy-max crazy-max added this to the v0.17.0 milestone Aug 6, 2024
@crazy-max crazy-max merged commit 5a50d13 into docker:master Aug 7, 2024
104 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants