Error: docker executable file not found when build in local cluster

[SergeyBear]

Actual behavior

Since skaffold 1.36.0 when following quick start guide on host with no docker installed, I get the following error when running skaffold dev or skaffold run:

$ skaffold dev
Listing files to watch...
 - test
Generating tags...
 - test -> test:3ad4ee077ebc361543f1f58b85a2c49a14d8a2bd4cb8ebc27b3052ee9254a21d
Checking cache...
 - test: Not found. Building
Starting build...
Found [minikube] context, using local docker daemon.
Building [test]...
Target platforms: [linux/amd64]
Cleaning up...
 - No resources found
build [test] failed: exec: "docker": executable file not found in $PATH. Docker build ran into internal error. Please retry.
If this keeps happening, please open an issue..

but skaffold build works file:

$ skaffold build
Generating tags...
 - test -> test:3ad4ee077ebc361543f1f58b85a2c49a14d8a2bd4cb8ebc27b3052ee9254a21d
Checking cache...
 - test: Not found. Building
Starting build...
Found [minikube] context, using local docker daemon.
Building [test]...
Sending build context to Docker daemon  126.8MB
...
Successfully tagged test:3ad4ee077ebc361543f1f58b85a2c49a14d8a2bd4cb8ebc27b3052ee9254a21d
Build [test] succeeded

running dev or run after build still does not work:

$ skaffold dev
Listing files to watch...
 - test
Generating tags...
 - test -> test:3ad4ee077ebc361543f1f58b85a2c49a14d8a2bd4cb8ebc27b3052ee9254a21d
Checking cache...
 - test: Not found. Building
Starting build...
Found [minikube] context, using local docker daemon.
Building [test]...
Target platforms: [linux/amd64]
Cleaning up...
 - No resources found
build [test] failed: exec: "docker": executable file not found in $PATH. Docker build ran into internal error. Please retry.
If this keeps happening, please open an issue..

while image present on minikube docker daemon registry:

$ minikube ssh
$ docker images | grep test
test                                      3ad4ee077ebc361543f1f58b85a2c49a14d8a2bd4cb8ebc27b3052ee9254a21d   66cc391e46f6   About a minute ago   640MB
test                                      66cc391e46f6d0506bde8ab33ae517ec088ce39002cba5858467ff7790bb9464   66cc391e46f6   About a minute ago   640MB
$

Force using docker API is not helping:

build:
  local:
    useDockerCLI: false

Expected behavior

Working with minikube without requiring installed docker - just like in 1.35.0 and earlier versions.

P.S. Fedora (and other rhel-based distros) replaced docker with podman and podman cli is not compatible with docker daemon, so cannot be used as docker cli replacement for skaffold.

Information

• Skaffold version: v2.0.3
• Operating system: Fedora 36
• Installed via: binary
• Contents of skaffold.yaml:

apiVersion: skaffold/v3
kind: Config
build:
  local:
    useDockerCLI: false
    push: false
  tagPolicy:
    inputDigest: {}
  artifacts:
    - image: test
manifests:
  kustomize:
    paths:
      - kustomize/base
deploy:
  kubectl: {}
  kubeContext: minikube

Debug output

In debug I see that it cannot find built image and tries to rebuild it, but with local docker daemon instead of remote:

$ skaffold dev -vdebug
DEBU[0000] skaffold API not starting as it's not requested  subtask=-1 task=DevLoop
INFO[0000] Skaffold &{Version:v2.0.3 ConfigVersion:skaffold/v4beta1 GitVersion: GitCommit:f5dee0f76014d4fb8df4eb89a845d5d45883ef96 BuildDate:2022-11-30T23:08:09Z GoVersion:go1.19.1 Compiler:gc Platform:linux/amd64 User:}  subtask=-1 task=DevLoop
INFO[0000] Loaded Skaffold defaults from "/home/user/.skaffold/config"  subtask=-1 task=DevLoop
DEBU[0000] config version out of date: upgrading to latest "skaffold/v4beta1"  subtask=-1 task=DevLoop
DEBU[0000] parsed 1 configs from configuration file /home/user/test/skaffold.yaml  subtask=-1 task=DevLoop
INFO[0000] map entry found when executing locate for &{test . <nil> {0xc0007331e0 <nil> <nil> <nil> <nil> <nil> <nil>} [] {[] []} []} of type *latest.Artifact and pointer: 824637555648  subtask=-1 task=DevLoop
INFO[0000] Using kubectl context: minikube               subtask=-1 task=DevLoop
DEBU[0000] getting client config for kubeContext: `minikube`  subtask=-1 task=DevLoop
INFO[0000] Using local-cluster=true from config          subtask=-1 task=DevLoop
DEBU[0000] getting client config for kubeContext: `minikube`  subtask=-1 task=DevLoop
DEBU[0000] Running command: [minikube version --output=json]  subtask=-1 task=DevLoop
DEBU[0000] Command output: [{"commit":"986b1ebd987211ed16f8cc10aed7d2c42fc8392f","minikubeVersion":"v1.28.0"}
]  subtask=-1 task=DevLoop
DEBU[0000] Running command: [/home/user/.local/bin/minikube docker-env --shell none -p minikube --user=skaffold]  subtask=-1 task=DevLoop
DEBU[0000] Command output: [MINIKUBE_ACTIVE_DOCKERD=minikube
DOCKER_TLS_VERIFY=1
DOCKER_HOST=tcp://192.168.39.234:2376
DOCKER_CERT_PATH=/home/user/.minikube/certs
]  subtask=-1 task=DevLoop
DEBU[0000] setting Docker user agent to skaffold-v2.0.3  subtask=-1 task=DevLoop
INFO[0000] Using minikube docker daemon at tcp://192.168.39.234:2376  subtask=-1 task=DevLoop
INFO[0000] manifests hydration will take place in /home/user/test/.kpt-pipeline  subtask=-1 task=DevLoop
DEBU[0000] Running command: [kubectl config view --minify -o jsonpath='{..namespace}']  subtask=-1 task=DevLoop
DEBU[0000] Command output: ['default']                   subtask=-1 task=DevLoop
DEBU[0000] Running command: [kubectl config view --minify -o jsonpath='{..namespace}']  subtask=-1 task=DevLoop
DEBU[0000] Command output: ['default']                   subtask=-1 task=DevLoop
DEBU[0000] Running command: [kubectl config view --minify -o jsonpath='{..namespace}']  subtask=-1 task=DevLoop
DEBU[0000] Command output: ['default']                   subtask=-1 task=DevLoop
DEBU[0000] Running command: [kubectl config view --minify -o jsonpath='{..namespace}']  subtask=-1 task=DevLoop
DEBU[0000] Command output: ['default']                   subtask=-1 task=DevLoop
DEBU[0000] Running command: [kubectl config view --minify -o jsonpath='{..namespace}']  subtask=-1 task=DevLoop
DEBU[0000] Command output: ['default']                   subtask=-1 task=DevLoop
DEBU[0000] Running command: [kubectl config view --minify -o jsonpath='{..namespace}']  subtask=-1 task=DevLoop
DEBU[0000] Command output: ['default']                   subtask=-1 task=DevLoop
DEBU[0000] Running command: [kubectl config view --minify -o jsonpath='{..namespace}']  subtask=-1 task=DevLoop
DEBU[0000] Command output: ['default']                   subtask=-1 task=DevLoop
DEBU[0000] CLI platforms provided: ""                    subtask=-1 task=DevLoop
DEBU[0000] getting client config for kubeContext: `minikube`  subtask=-1 task=DevLoop
DEBU[0000] platforms detected from active kubernetes cluster nodes: "linux/amd64"  subtask=-1 task=DevLoop
DEBU[0000] platforms selected for artifact "test": "linux/amd64"  subtask=-1 task=DevLoop
DEBU[0000] Using builder: local                          subtask=-1 task=DevLoop
INFO[0000] build concurrency first set to 1 parsed from *local.Builder[0]  subtask=-1 task=DevLoop
INFO[0000] final build concurrency value is 1            subtask=-1 task=DevLoop
Listing files to watch...
 - test
DEBU[0000] Found dependencies for dockerfile: [{app /opt/app true 38 38}]  subtask=-1 task=DevLoop
DEBU[0000] Executing template &{envTemplate 0xc00028b200 0xc000398050  } with environment map[BASH_FUNC_which%%:() {  ( alias;
...
INFO[0000] List generated in 66.844078ms                 subtask=-1 task=DevLoop
Generating tags...
 - test -> DEBU[0000] Found dependencies for dockerfile: [{app /opt/app true 38 38}]  subtask=-1 task=DevLoop
test:3ad4ee077ebc361543f1f58b85a2c49a14d8a2bd4cb8ebc27b3052ee9254a21d
INFO[0001] Tags generated in 589.581659ms                subtask=-1 task=Build
Checking cache...
DEBU[0001] Could not import artifact from Docker, building instead (import of missing images disabled)  subtask=-1 task=Build
 - test: Not found. Building
INFO[0001] Cache check completed in 575.640977ms         subtask=-1 task=Build
Starting build...
Found [minikube] context, using local docker daemon.
Building [test]...
Target platforms: [linux/amd64]
DEBU[0001] setting DOCKER_BUILDKIT=1 for docker build for artifact "test" since it targets platform "linux/amd64"  subtask=test task=Build
DEBU[0001] Running command: [docker build . --file /home/user/test/Dockerfile -t test:3ad4ee077ebc361543f1f58b85a2c49a14d8a2bd4cb8ebc27b3052ee9254a21d --platform linux/amd64]  subtask=test task=Build
Cleaning up...
DEBU[0001] Running command: [kubectl --context minikube delete --ignore-not-found=true --wait=false -f -]  subtask=-1 task=DevLoop
 - No resources found
INFO[0002] Cleanup completed in 64.168567ms              subtask=-1 task=DevLoop
DEBU[0002] Running command: [tput colors]                subtask=-1 task=DevLoop
DEBU[0002] Command output: [256
]                        subtask=-1 task=DevLoop
build [test] failed: exec: "docker": executable file not found in $PATH. Docker build ran into internal error. Please retry.
If this keeps happening, please open an issue..
DEBU[0002] exporting metrics                             subtask=-1 task=DevLoop
DEBU[0002] error uploading metrics: rpc error: code = Unauthenticated desc = Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.  subtask=-1 task=DevLoop
DEBU[0002] writing to file /home/user/.skaffold/metrics instead  subtask=-1 task=DevLoop

[SergeyBear]

Just found that specifying platform in build section also fails the build, so using cli probably goes from there.

With platform:

build:
  ...
  platforms:
    - linux/amd64

build fail:

$ skaffold build
Generating tags...
 - test -> test:3ad4ee077ebc361543f1f58b85a2c49a14d8a2bd4cb8ebc27b3052ee9254a21d
Checking cache...
 - test: Not found. Building
Starting build...
Found [minikube] context, using local docker daemon.
Building [test]...
Target platforms: [linux/amd64]
exec: "docker": executable file not found in $PATH. Docker build ran into internal error. Please retry.
If this keeps happening, please open an issue..

Without platform:

build:
  ...
  # platforms:
  #   - linux/amd64

build success:

Generating tags...
 - test -> test:3ad4ee077ebc361543f1f58b85a2c49a14d8a2bd4cb8ebc27b3052ee9254a21d
Checking cache...
 - test: Found Locally

[onimsha]

I got same problem. Already opened an issue a while back. #7883

[wangning0]

I got same problem...

[gsquared94]

you're right that it's failing due to specifying the platforms stanza in the skaffold.yaml file, that requires building using docker CLI.
You should be able to circumvent this by removing the platforms stanza and running skaffold dev --check-cluster-node-platforms=false