Implement new build flag --docker-host

[matejvasek]

Summary

Introducing new build flag: --docker-host.
The purpose of the flag is to expose custom docker deamon to build
container. This is useful when running docker daemon on non standard
location e.g. "tcp://localhost:1234", or if resulting image should be
stored in different daemon than you are running locally.

[matejvasek]

Using TCP via --net=host and envvar might be advantageous when using on system with SELinux since SELinux doesn't like mounting things into container without proper policies set.

[matejvasek]

This is also of interest for #966.

[matejvasek]

Docker doc also mentions fd:// protocol but I never used it.

[codecov]

Codecov Report

Merging #988 (fcd3103) into main (650ce05) will increase coverage by 0.03%.
The diff coverage is 97.62%.

@@            Coverage Diff             @@
##             main     #988      +/-   ##
==========================================
+ Coverage   80.35%   80.38%   +0.03%     
==========================================
  Files         130      130              
  Lines        7964     7976      +12     
==========================================
+ Hits         6399     6411      +12     
  Misses       1147     1147              
  Partials      418      418              

Flag	Coverage Δ
os_linux	79.70% <97.06%> (+0.02%)	⬆️
os_macos	77.15% <97.06%> (+0.04%)	⬆️
os_windows	80.30% <97.62%> (+0.03%)	⬆️
unit	79.75% <97.06%> (+0.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

[jromero]

Hi @matejvasek, thanks for this PR. Just wanted to give you an update and heads up. Given the holiday's it'll probably take a little bit longer to get through the process. I'm setting up an environment to test this. I have a few comments I'll add to the PR but it's great to see this feature finally coming in.

Adding some context for others, pack does support DOCKER_HOST for managing the containers it uses to execute the lifecycle (aka build process). The problem this PR is attempting to address is the fact that we currently blindly try to mount the docker socket when --publish is false. This ultimately fails when the lifecycle tries to use the daemon since it doesn't have the DOCKER_HOST information. In other words, this PR provides the DOCKER_HOST information to the lifecycle.

[jromero]

Update: So I ran into a case where this conflicts with the use of Docker Desktop. Currently the mount /var/run/docker.sock:/var/run/docker.sock works when the remote docker daemon is "Docker Desktop" because it mounts the internal socket from the VM of the daemon host (not the client).

The proposed solution in this PR currently adds complexity that isn't quite a drop-in replacement. For example, if you use a tcp://docker:1234 remote daemon and docker hostname is local to the client via /etc/hosts the lifecycle container wouldn't be able to resolve the hostname.

I do wonder if this solution could be optionally applied via detecting the host daemon process (docker desktop vs podman) or (less preferable) by using an optional flag/configuration element. Overall, I think it's going in the right direction but we'll want to make sure it's not disruptive out of the gate.

[dfreilich]

Thank you for the PR! I have a few reservations about the PR as is:

1. Tests added for these cases would help the robustness
2. We'd probably want to thread the logger through the provider, so we don't have to write directly to errorWriter

I'm happy to take care of both of those in a polish PR, but I do want to clarify the specific logical questions I had – notably, which OS's we think we need to set host (so we can have an appropriate comment clarifying what we're doing, and setting provider.ctrConf.Env even for non linux environments.

Definitely hoping to get this in in the next few days!

[matejvasek]

The proposed solution in this PR currently adds complexity that isn't quite a drop-in replacement. For example, if you use a tcp://docker:1234 remote daemon and docker hostname is local to the client via /etc/hosts the lifecycle container wouldn't be able to resolve the hostname.

@jromero I know that's problematic, on Linux this can be solved by using --network=host I don't know how to fix that on other OSes.

[micahyoung]

To summarize my thinking, I love the idea of being able to specify an alternative to the default bind-mounted socket. I'm a bit concerned with this implementation right now because it's very unencrypted-TCP-on-Linux oriented and breaks the currently working, TCP-over-mTLS which is platform agnostic and much safer for general use.

I wonder, what if this were done with pack build option like --docker-host=
which, if specified, would always disable the default binding and sets a "universal" docker host for pack and lifecycle, supporting values like:

Use-case 1: localhost-over-unencrypted-TCP

Just like pack's current behavior (respect DOCKER_HOST) but also copy it to down to lifecycle

export DOCKER_HOST=tcp://localhost:2375
pack build my-app --docker-host=inherit --network=host

# equivalent
pack build my-app --docker-host=tcp://localhost:2375 --network=host

Use-case 2: Non-standard socket location

This would bind-mount anything with a prefix of unix: (or npipe: for Windows) and use it as pack and lifecycle's DOCKER_HOST (though maybe using the standard socket path in the container). This should enable rootless docker (which as of last year, did not work due to the hard-coded host socket path)

pack build my-app --docker-host=unix:///run/user/1001/docker.sock

Use-case 3: mTLS-secured daemon

Parse any TCP URI for optional cert options (syntax could be tweaked but a cert paths would also set DOCKER_TLS_VERIFY and DOCKER_CERT_PATH for lifecycle). More details on use-case.

pack build my-app --docker-host=tcp://my-secure-host:2376?tlscertpath=/docker-certs --volume $HOME/.docker:/docker-certs

We could start with just Use-case 1 which could expand into the other cases with future contributions.

[matejvasek]

Thanks for the feedback. I hope I'll get back to this next week.

[dfreilich]

Hey @matejvasek ! Just wanted to check on in this – this is something we're definitely interested in enabling, and it would be great to get this done. Do you think you'll have some time for this over the next few weeks? If not, I know @micahyoung had mentioned that he had some ideas for doing it as well.

[matejvasek]

@dfreilich sorry for not getting to this earlier, I had to fix some podman issues to make this feature useful for me. I've just pushed new commit adding new docker-host flag as @micahyoung suggested. Tests are TODO. It should support first two use-cases, but not the third one.

[micahyoung]

Apologies, but I realize I missed one aspect of the workflow in my earlier comment, that the previous behavior needs to be the new default (namely, when --docker-host is not provided, don't pass through DOCKER_HOST env vars into the container and always using a bind mount). Unfortunately, inherit isn't sufficient for a backward compatible default.

I was able to fix this in your PR by setting the default value for the flag to "" instead of "inherit". I'd be happy with this change for now and if needs any further cleanup, I can do another pass when I add in the TLS behavior.

[matejvasek]

@micahyoung this is odd build / test (windows-lcow) fails after minor change.

[matejvasek]

@micahyoung could you please try restarting the action?

[matejvasek]

I'll have to squash the commits anyway.

[micahyoung]

Thank you for the quick implementation and changes. Looking forward to using the feature. It appears the last run went green as well.

[matejvasek]

squashed

[dfreilich]

Thank you so much for doing this, this looks really good, and really close! I'm sorry for sending this back, just a few small things I'd like to see added to this.

[matejvasek]

@dfreilich PTAL

[dfreilich]

LGTM

Thank you so much for this, this ended up being considerably more complex than we initially thought, and I appreciate all the work that went into this!

@micahyoung I'll give you time to check it out as well (if you have the time).

[matejvasek]

@micahyoung what is your use-case for this? Are you pushing images into remote daemon?

[micahyoung]

LGTM. Great contribution, many thanks!

[micahyoung]

@micahyoung what is your use-case for this? Are you pushing images into remote daemon?

Yeah, I am using Docker Desktop Windows daemons (and recently Docker Desktop MacOS/ARM64 daemons) on local/remote VMs mostly, with the daemon API either unencrypted or secured with mTLS, but running pack commands from my non-Windows Workstation. I develop buildpacks and stacks mostly for Windows OCI images (and contribute support for this on pack/lifecycle). Having various daemons (Linux/Windows, ARM/AMD64, podman/docker) and just swapping the DOCKER_HOST makes it easy to try out various combinations of things against lots of different daemons:

My current workflow, before this feature, was based on swapping around the DOCKER_HOST/DOCKER_CERT_PATH:

• Targeting Windows daemon insecurely on local VM

  source ~/.docker/docker-windows-local/env.sh
  
  docker build ...
  
  # contents of env.sh
  export DOCKER_HOST=tcp://<local VM IP>:2375
  unset DOCKER_TLS_VERIFY
  unset DOCKER_CERT_PATH
  

• Targeting Windows daemon securely over an untrusted network:

  source ~/.docker/docker-windows-gcp/env.sh
  
  docker build ...
  
  # contents of env.sh
  export DOCKER_HOST=tcp://<windows daemon IP>:2376
  export DOCKER_TLS_VERIFY=1
  export DOCKER_CERT_PATH=~/.docker/docker-windows-gcp/  # self-signed client/server certs
  

With the functionality of this feature, I should be able to now do this from the command line:

• Targeting Windows daemon on local VM

  docker build --docker-host=tcp://<local VM IP>:2375 ...
  

• Targeting Windows daemon on an untrusted network:
  ... until, the tls* options for --docker-host=tcp://..., are supported, I'll still need to connect to secured TLS daemons with:

  source ~/.docker/docker-windows-gcp/env.sh
  
  docker build --docker-host=""  ...  # or unset
  

  ... but eventually, with TLS cert options (syntax TBD)

  docker build --docker-host=tcp://<windows daemon IP>:2376 --docker-host-options=tlscertpath=~/.docker/docker-windows-gcp,tlsverify=1
  

  There's a few details here that may be tricky: how to copy in certs in a way that only lifecycle can access them but not buildpacks (maybe root/Administrator-only read permissions?). Or perhaps an option to only use --docker-host="tcp://..."
  and certs for pack, but continue to bind-mount socket in the container for lifecycle.

But overall, this feature is a great step to making the implicit DOCKER_HOST functionality that I've been relying on become more explicit, while enabling more rootless daemon options which is awesome.