Local exporter: Export individual files

[jleben]

Is there a way to export only parts of the output filesystem to local filesystem?

I haven't found a buildctl option to do that.

I guess I could theoretically define a graph that ends with a scratch state and mounts parts of another state. However, the client library doesn't seem to provide AddMount for State, but only for ExecState, and I don't see a trivial way to form the latter from the former. In any case, such an additional mounting step would seem like an avoidable inconvenience to me...

[jleben]

I figured out I can copy files from an arbitrary state into a scratch state this way (where source is the state I want to copy from, and it has the command cp available):

out := llb.Scratch()
copy := source.
    Run(llb.Shlex("cp /something /output")).
    AddMount("/output", out)

This mimics the approach of the copy function in this example: https://github.com/moby/buildkit/blob/master/examples/buildkit0/buildkit.go

Perhaps this copying could be avoided by extending the local exporter, or by extending buildctl build --local to support mounting writable local directories in containers?

[tonistiigi]

I'm fine with adding filter option to the local exporter. The copy example will become simpler and more powerful when #809 lands.

[thaJeztah]

@tonistiigi @crazy-max @jedevc wondering how complicated this feature is to implement.

I love the docker build --output option, but find myself having to explain "but you need to have a FROM scratch stage to collect what you want to export" every time I explain the feature to users.

While I still think there's many scenarios where that's a "good practice", there's also many scenarios where it would be great to only export specific path(s) from a stage, without having to define a "special" stage for that.

So (for example), if my stage has build-artifacts in /app/build, I could use something like;

docker build --output src=/app/build,dst=./build .

That stage may (potentially) be both for an image (so perhaps I could --tag the build as well), as well as to "export".

[alexgwolff]

Hi @tonistiigi, are you okay with adding the src parameter? Also, should we support this parameter when the output is tar as well? Do you have any concerns or suggestions about making this change? Thanks!

docker buildx build --output type=local,dest=./,src=./my-path-inside-container
docker buildx build --output type=tar,dest=./,src=./my-path-inside-container

[tonistiigi]

Yes, looks fine. Both tar and local.

[fenollp]

How about glob patterns on source? E.g.

FROM scratch AS out-0d0258aff29ab4b2
COPY --from=dep-l-anstyle-1.0.10-0d0258aff29ab4b2 /tmp/clis-cargo-udeps_0-1-55/release/deps/*-0d0258aff29ab4b2* /

cf https://docs.docker.com/reference/dockerfile/#pattern-matching-1

Would this be possible? (note the single quotes)

docker buildx build --output type=local,dest=.,src='/tmp/clis-cargo-udeps_0-1-55/release/deps/*-0d0258aff29ab4b2*'