FR: add support for `minikube kubectl` to skaffold (currently errors when `kubectl` not present but `minikube kubectl` available)

[kal93]

tldr: skaffold doesn't recognize minikube's kubectl add on

Expected behavior

Should start dev mode deployment when skaffold dev is run with minikube

Actual behavior

skaffold dev terminates with below error

Starting deploy...
WARN[0000] unable to get kubectl client version: starting command kubectl version --client -ojson: exec: "kubectl": executable file not found in $PATH  subtask=-1 task=DevLoop
kubectl client version: unknown
couldn't get kubectl minor version: strconv.Atoi: parsing "unknown": invalid syntax
Cleaning up...
WARN[0000] deployer cleanup:couldn't get kubectl minor version: strconv.Atoi: parsing "unknown": invalid syntax. Please install kubeclt via https://kubernetes.io/docs/tasks/tools/install-kubectl.  subtask=-1 task=DevLoop
couldn't get kubectl minor version: strconv.Atoi: parsing "unknown": invalid syntax. Please install kubeclt via https://kubernetes.io/docs/tasks/tools/install-kubectl.

Information

• Skaffold version: v1.35.0
• Operating system: Ubuntu 21
• Installed via: skaffold.dev
• Contents of skaffold.yaml:

# Used only for dev ENV.Watches the config files in infra/k8s directory and deploys automatically.
apiVersion: skaffold/v2beta26
kind: Config
deploy:
  kubectl:
    manifests:
      - ./infra/k8s/*
build:
  local:
    push: false
  artifacts:
    - image: johndoeups/auth
      context: auth
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: 'src/**/*.ts'
            dest: .

Steps to reproduce the behavior

1. Run the above skaffold.yaml with skaffold dev

-vdebug logs

DEBU[0000] skaffold API not starting as it's not requested  subtask=-1 task=DevLoop
INFO[0000] Skaffold &{Version:v1.35.0 ConfigVersion:skaffold/v2beta26 GitVersion: GitCommit:e9c900fe353d50a536de042eca844e9ca4ed2ac7 BuildDate:2021-11-16T20:24:22Z GoVersion:go1.14.14 Compiler:gc Platform:linux/amd64 User:}  subtask=-1 task=DevLoop
INFO[0000] Loaded Skaffold defaults from "/home/9L/.skaffold/config"  subtask=-1 task=DevLoop
DEBU[0000] parsed 1 configs from configuration file /home/9L/Repos/TicketingApp/skaffold.yaml  subtask=-1 task=DevLoop
INFO[0000] Using kubectl context: custom                 subtask=-1 task=DevLoop
INFO[0000] Using local-cluster=true from config          subtask=-1 task=DevLoop
DEBU[0000] Running command: [minikube version --output=json]  subtask=-1 task=DevLoop
DEBU[0000] Command output: [{"commit":"76b94fb3c4e8ac5062daf70d60cf03ddcc0a741b","minikubeVersion":"v1.24.0"}
]  subtask=-1 task=DevLoop
DEBU[0000] Minikube cluster detected: cluster certificate for context "custom" found inside the minikube directory  subtask=-1 task=DevLoop
DEBU[0000] Running command: [/usr/bin/minikube docker-env --shell none -p custom --user=skaffold]  subtask=-1 task=DevLoop
DEBU[0000] Command output: [MINIKUBE_ACTIVE_DOCKERD=custom
DOCKER_TLS_VERIFY=1
DOCKER_HOST=tcp://192.168.58.2:2376
DOCKER_CERT_PATH=/home/9L/.minikube/certs
]  subtask=-1 task=DevLoop
DEBU[0000] setting Docker user agent to skaffold-v1.35.0  subtask=-1 task=DevLoop
INFO[0000] Using minikube docker daemon at tcp://192.168.58.2:2376  subtask=-1 task=DevLoop
DEBU[0000] Using builder: local                          subtask=-1 task=DevLoop
INFO[0000] build concurrency first set to 1 parsed from *local.Builder[0]  subtask=-1 task=DevLoop
INFO[0000] final build concurrency value is 1            subtask=-1 task=DevLoop
Listing files to watch...
 - johndoeups/auth
DEBU[0000] Found dependencies for dockerfile: [{package.json /usr/apps/ticketing/auth true 8 8} {. /usr/apps/ticketing/auth true 14 14}]  subtask=-1 task=DevLoop
DEBU[0000] Skipping excluded path: node_modules         
INFO[0000] List generated in 1.26141ms                   subtask=-1 task=DevLoop
Generating tags...
 - johndoeups/auth -> DEBU[0000] Running command: [git describe --tags --always]  subtask=-1 task=Build
DEBU[0000] generating tag: unable to find git commit: running [git describe --tags --always]
 - stdout: ""
 - stderr: "fatal: not a git repository (or any of the parent directories): .git\n"
 - cause: exit status 128  subtask=johndoeups/auth task=Build
DEBU[0000] Using a fall-back tagger                      subtask=johndoeups/auth task=Build
johndoeups/auth:latest
Some taggers failed. Rerun with -vdebug for errors.
INFO[0000] Tags generated in 1.013291ms                  subtask=-1 task=Build
Checking cache...
 - johndoeups/auth: Found Locally
INFO[0000] Cache check completed in 2.87332ms            subtask=-1 task=Build
Tags used in deployment:
 - johndoeups/auth -> johndoeups/auth:b9f4e8440cf450ccfd6a4acce36464a60f1dfc78fb37d6c5b7990d56320f9651
DEBU[0000] Local images can't be referenced by digest.
They are tagged and referenced by a unique, local only, tag instead.
See https://skaffold.dev/docs/pipeline-stages/taggers/#how-tagging-works  subtask=-1 task=Deploy
Starting deploy...
DEBU[0000] getting client config for kubeContext: `custom`  subtask=-1 task=DevLoop
DEBU[0000] Running command: [kubectl version --client -ojson]  subtask=0 task=Deploy
WARN[0000] unable to get kubectl client version: starting command kubectl version --client -ojson: exec: "kubectl": executable file not found in $PATH  subtask=-1 task=DevLoop
kubectl client version: unknown
couldn't get kubectl minor version: strconv.Atoi: parsing "unknown": invalid syntax
Cleaning up...
WARN[0000] deployer cleanup:couldn't get kubectl minor version: strconv.Atoi: parsing "unknown": invalid syntax. Please install kubeclt via https://kubernetes.io/docs/tasks/tools/install-kubectl.  subtask=-1 task=DevLoop
DEBU[0000] Running command: [tput colors]                subtask=-1 task=DevLoop
DEBU[0000] Command output: [256
]                        subtask=-1 task=DevLoop
couldn't get kubectl minor version: strconv.Atoi: parsing "unknown": invalid syntax. Please install kubeclt via https://kubernetes.io/docs/tasks/tools/install-kubectl.

Bash alias appears to be correct as I get correct output for kubectl --help as mentioned in #4576.

kubectl controls the Kubernetes cluster manager.

 Find more information at: https://kubernetes.io/docs/reference/kubectl/overview/

Basic Commands (Beginner):
  create        Create a resource from a file or from stdin
  expose        Take a replication controller, service, deployment or pod and expose it as a new Kubernetes service
  run           Run a particular image on the cluster
  set           Set specific features on objects

Basic Commands (Intermediate):
  explain       Get documentation for a resource
  get           Display one or many resources
  edit          Edit a resource on the server
  delete        Delete resources by file names, stdin, resources and names, or by resources and label selector

Deploy Commands:
  rollout       Manage the rollout of a resource
  scale         Set a new size for a deployment, replica set, or replication controller
  autoscale     Auto-scale a deployment, replica set, stateful set, or replication controller

Cluster Management Commands:
  certificate   Modify certificate resources.
  cluster-info  Display cluster information
  top           Display resource (CPU/memory) usage
  cordon        Mark node as unschedulable
  uncordon      Mark node as schedulable
  drain         Drain node in preparation for maintenance
  taint         Update the taints on one or more nodes

Troubleshooting and Debugging Commands:
  describe      Show details of a specific resource or group of resources
  logs          Print the logs for a container in a pod
  attach        Attach to a running container
  exec          Execute a command in a container
  port-forward  Forward one or more local ports to a pod
  proxy         Run a proxy to the Kubernetes API server
  cp            Copy files and directories to and from containers
  auth          Inspect authorization
  debug         Create debugging sessions for troubleshooting workloads and nodes

Advanced Commands:
  diff          Diff the live version against a would-be applied version
  apply         Apply a configuration to a resource by file name or stdin
  patch         Update fields of a resource
  replace       Replace a resource by file name or stdin
  wait          Experimental: Wait for a specific condition on one or many resources
  kustomize     Build a kustomization target from a directory or URL.

Settings Commands:
  label         Update the labels on a resource
  annotate      Update the annotations on a resource
  completion    Output shell completion code for the specified shell (bash or zsh)

Other Commands:
  api-resources Print the supported API resources on the server
  api-versions  Print the supported API versions on the server, in the form of "group/version"
  config        Modify kubeconfig files
  plugin        Provides utilities for interacting with plugins
  version       Print the client and server version information

Usage:
  kubectl [flags] [options]

Use "kubectl <command> --help" for more information about a given command.
Use "kubectl options" for a list of global command-line options (applies to all commands).

Am I missing something?

[kal93]

@aaron-prindle any idea when to expect a fix for this?

[aaron-prindle]

Hi @kal93. The issue here is that currently skaffold expects a binary named "kubectl" to be in the path:
https://github.com/GoogleContainerTools/skaffold/blob/main/pkg/skaffold/kubectl/cli.go#L59-L63

// Command creates the underlying exec.CommandContext. This allows low-level control of the executed command.
func (c *CLI) Command(ctx context.Context, command string, arg ...string) *exec.Cmd {
	args := c.args(command, "", arg...)
	return exec.CommandContext(ctx, "kubectl", args...)
}

Skaffold uses exec.Command under the hood to call kubectl and this does not respect bash aliases as bash is not used to execute this. I don't believe there is a simple workaround for this at the moment, I can take this on as a TODO for the next sprint. I will update the bug title here to be a feature request for adding support for minikube kubectl for skaffold.

[luukvhoudt]

This work around I can confirm:

#4576 (comment)

Add this in a file named kubectl to a $PATH directory and chmod +x it.