Homebrew installed Skaffold gives "invalid Host header" error

[habuma]

Expected behavior

skaffold dev, skaffold run, and skaffold build should work as expected with the Homebrew-installed Skaffold without undue error.

Actual behavior

When using Skaffold installed on Mac via Homebrew, those Skaffold commands result in an error like:

build [habuma/hello] failed: executing lifecycle: http: invalid Host header

I experienced this error on 3 different MacOS machines (one Apple M1, one Apple M1 Pro, and one Intel) and across three different cluster types (Kind, Docker Desktop Kubernetes, and K3D).

After uninstalling Skaffold via Homebrew, I reinstalled from a downloaded binary. It now works as expected.

Information

• Skaffold version: I tried this with several Skaffold versions, but most recently v2.6.3
• Operating system: MacOS Ventura 13.4.1
• Installed via: Homebrew
• Contents of skaffold.yaml:

apiVersion: skaffold/v4beta6
kind: Config
metadata:
  name: hello-k-s
build:
  artifacts:
    - image: habuma/hello-k8s
      buildpacks:
        builder: gcr.io/buildpacks/builder:v1
        env:
          - "GOOGLE_RUNTIME_VERSION=17"
        trustBuilder: true
manifests:
  rawYaml:
    - target/classes/META-INF/dekorate/kubernetes.yml

(That said, I tried this with several different variations of that skaffold.yaml file and with several different variations of a deployment manifest that I know have worked in the past.)

Steps to reproduce the behavior

1. Install Skaffold via Homebrew
2. skaffold dev, skaffold run, or skaffold build
3. See error

I no longer have access to such logs after reinstalling Skaffold from the downloaded binary. If deemed necessary, I can reinstall via Homebrew on one of my machines to retrieve debug logs. However, I viewed those logs myself and found nothing of great interest leading up to that error.

What I did discover, however, was that the underlying "invalid Host header" error seems to be coming from the Docker client. It might be related to docker/cli#4437 and/or golang/go#61431. I'll leave the diagnosis to the Skaffold team, but I speculate that the Homebrew version is built against a buggy version of the client, whereas the downloaded binary is built against a fixed version.

[habuma]

Followup: For grins I reinstalled via Homebrew and was able to pull those logs, attached here.

skaffold_debug_logs.txt

Also, I realized that the Homebrew version is v2.6.3 whereas the downloaded binary was v2.6.0. Also, the binary size between each is significantly different:

• Locally installed binary (v2.6.0): 88128288
• Homebrew installed (v2.6.3): 100477152

I'm not sure that the size of the binary necessarily matters, but it does suggest that there is a significant difference between these two versions.

[ericzzzzzzz]

Hi, @habuma , Thank you for filing this issue.

we don't have direct control over the skaffold binary from homebrew, the one in homebrew is not built with the go sdk we use in our ci, instead it is built with a different go sdk version https://github.com/Homebrew/homebrew-core/pull/138618/files and homebrew team currently does the maintenance , we'll reach out to them to see what we can do!

[ericzzzzzzz]

I was going to send a email to a member of homebrew, but I found this https://github.com/orgs/Homebrew/discussions/2790#discussioncomment-1970688 , this seems that there is nothing we can do about it the binary discrepancy between the from homebrew and the one from gcs. sorry.