Skaffold does not work when using remote docker over ssh

[agorgl]

Expected behavior

Using skaffold in an environment where DOCKER_HOST=ssh://somehost is defined should work, as this is a supported docker transport method.

Actual behavior

Skaffold tries to connect over http to the given host, ignoring the 'ssh://' part

Information

• Skaffold version: v1.22.0
• Operating system: Windows
• Installed via: Scoop

Steps to reproduce the behavior

1. Run docker on a remote host that you can also connect with ssh
2. Set environment variable DOCKER_HOST=ssh://yourhost
3. Run skaffold dev on a project

[briandealwis]

I don't see any references within the Docker client code (github.com/docker/docker/client/) and the references I found within Moby indicated that it was part of buildkit (github.com/moby/buildkit/client/llb/exec.go). Need to investigate whether this is supported upstream.

You might be able to get this to work by setting build.local.useDockerCLI: true.

[agorgl]

Using:

build:
  local:
    useDockerCLI: true

results in the same behavior. Keep a note that regular docker cli works as it shoulds with DOCKER_HOST set as described above.

[Dev-Time]

I believe I'm getting the same issue. Here's the actual error:

getting imageID for <docker image>: error during connect: Get "http://<username>%40<ssh host>/v1.24/images/<docker image>/json": dial tcp: lookup <username>@<ssh host>: no such host. Docker build ran into internal error. Please retry.
If this keeps happening, please open an issue..

Im getting this error with all combinations of build.local.useBuildkit and build.local.useDockerCLI

[tboevil]

Add -H tcp://<your-remote-ip> to /usr/lib/systemd/system/docker.service, and use env DOCKER_HOST=tcp://<your-remote-ip>:2375 skaffold build -v debug instead of skaffold build -v debug, it's work to me.

It seems to have something to do with this

Maybe need a PR

[agorgl]

This exposes docker deamon over tcp which is massively insecure.

[tboevil]

This exposes docker deamon over tcp which is massively insecure.

yes，it is only suitable for pure intranet environment

[agorgl]

Still, control over ssh channel is a way better alternative (for systems that are behind firewalls etc exposing only the ssh access) and should be a first priority support over tcp daemon

[weltonrodrigo]

Add -H tcp://<your-remote-ip> to /usr/lib/systemd/system/docker.service, and use env DOCKER_HOST=tcp://<your-remote-ip>:2375 skaffold build -v debug instead of skaffold build -v debug, it's work to me.

It seems to have something to do with this

Maybe need a PR

As you may know, running docker with tcp basically GIVES ANYONE WHO CAN REACH THIS PORT ROOT ACCESS TO THE MACHINE. https://docs.docker.com/go/attack-surface/

That being said:

On the remote machine:

Add an override to your systemd docker configuration:

sudo systemctl edit docker

This will open the editor, edit it so the first lines look like:

### Editing /etc/systemd/system/docker.service.d/override.conf
### Anything between here and the comment below will become the new contents of the file

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 --containerd=/run/containerd/containerd.sock

### Lines below this comment will be discarded

Restart your docker service on the remote machine:

sudo systemctl restart docker

On your machine:

Let's say the remote machine is 192.168.0.20:

docker context create remote \
   --default-stack-orchestrator=swarm \
   --docker host=tcp://192.168.0.20:2375

Now, when you want to use the remote machine, switch context:

docker context use remote

When you want to use local docker:

docker context use default

Using skaffold with the remote docker:

Unfortunately, skaffold won't recognize the context docker is in, so you still will have to inform that in a environment variable.

DOCKER_HOST=tcp://192.168.0.20:2375 skaffold build

[stephenstubbs]

Having the same issue using docker in a LXD container and connecting to it from CLI on the host. I can just disable TLS but definitely would be a security problem if accessing remotely.

[kubaw]

You can forward socket file using ssh and -L option. For example:

ssh -L $HOME/remote-docker.sock:/var/run/docker.sock user@docker-remote-host.example.com

Where /var/run/docker.sock is the dockerd socket file on remote machine docker-remote-host.example.com

Then you can use this local socket to forward queries to remote dockerd:

DOCKER_HOST=unix://$HOME/remote-docker.sock skaffold dev