HTTP Proxy support is broken after upgrade to Version 17.12.0-ce-mac46 (21698)

[alvarow]

Expected behavior

Docker pull from internal or external registries work through HTTP proxy that does SSL Interception.

Actual behavior

If the HTTP and HTTPS Proxy field are set on Docker Preferences, then the internal registry fails with ERROR: Get https://dtr.cdl.es.ad.xxx.com/v2/: net/http: TLS handshake timeout message, but works for external registries (i.e. AWS ECR or Docker Hub).

If the HTTP Proxy preference is removed (but HTTPS is left) then the internal registry works (its SSL certificate is signed by the same internal CA that signs the SSL Intercepted requests through the proxy), but external registries fail to load with Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Information

• Full output of the diagnostics from "Diagnose & Feedback" in the menu

Diagnostic ID 94B69674-4A18-4FC4-A9E4-2644691FA387

Docker for Mac: version: 17.12.0-ce-mac46 (a61e84b8bca06b1ae6ce058cdd7beab1520ad622)
macOS: version 10.12.6 (build: 16G1036)
logs: /tmp/94B69674-4A18-4FC4-A9E4-2644691FA387/20180109-215355.tar.gz
[OK]     db.git
[OK]     vmnetd
[OK]     dns
[OK]     driver.amd64-linux
[OK]     virtualization VT-X
[OK]     app
[OK]     moby
[OK]     system
[OK]     moby-syslog
[OK]     kubernetes
[OK]     env
[OK]     virtualization kern.hv_support
[OK]     slirp
[OK]     osxfs
[OK]     moby-console
[OK]     logs
[OK]     docker-cli
[OK]     menubar
[OK]     disk

Steps to reproduce the behavior

1. From a network that requires authenticated proxy usage with SSL Interception (i.e. BlueCoat), try to perform docker login or docker pull.

[djs55]

Thanks for your report.

I've not been able to reproduce this myself yet, but I've found a bug which I suspect could be the cause of this-- in the proxy configuration code we conflated the settings for the HTTP and HTTPS proxies, leading to unexpected results in some environments.

I've made a prototype fix. If you would like to try it I've made an experimental build: https://download-stage.docker.com/mac/pr/21758/Docker.dmg -- note this build is suitable for testing only. The build is version

Version 17.12.1-ce-rc1-mac49 (21758)
Channel: pr
f007f92cf0

If it offers an auto-update, don't accept it since it will probably update to another experimental build without the fix.

If this doesn't work, could you let me know a little bit more about your proxies? Do you have different URLs for HTTP and HTTPS?

[alvarow]

Thanks, I will give it a try in a few minutes.

I have the same URL for HTTP and HTTPS, I don't have a whole lot of details, I simply know it is a Bluecoat cluster that requires authentication and performs SSL Interception (DLP checks etc).

This has been working for well over a year, just now, whatever the changes are seems to have affected it.

Thanks for the quick reply.

[alvarow]

No joy :-(

regulya@ROSELCDV0001LHJ:~$ docker login dtr.cdl.es.ad.xxx.com
Username: adpe
Password:
Error response from daemon: Get https://dtr.cdl.es.ad.xxx.com/v2/: net/http: TLS handshake timeout
regulya@ROSELCDV0001LHJ:~$ docker pull redis:alpine
alpine: Pulling from library/redis
605ce1bd3f31: Pull complete
d12d456b03b7: Pull complete
b6e6ee69c5e2: Pull complete
813ea0248ee4: Pull complete
62208b7f07e1: Pull complete
35b7abefa067: Pull complete
Digest: sha256:a5fd9334b28d6afdc8b84aad8a01056d8d1d849cb5a744919caf40d5b57597b2
Status: Downloaded newer image for redis:alpine

The proxy works, I can pull things from the external registry, just not the internal one, seems it won't trust the internal CA even with its certificate being present on Keychain.

docker info however gives me conflicting information about the proxies:

---snip---
HTTP Proxy: docker.for.mac.http.internal:3128
HTTPS Proxy: docker.for.mac.http.internal:3129
---snip---

Which most certainly is completely different that the proxy configured on Preferences.

[alvarow]

DiagnosticID is 94B69674-4A18-4FC4-A9E4-2644691FA387

[djs55]

Thanks for giving it a go. I'll investigate more tomorrow.

The docker.for.mac.http.internal name resolves to an internal TCP proxy which connects to the proxy configured in Preferences. Or at least that is the theory :-)

[rubyisbeautiful]

This also affected me and some colleagues. We use the same proxy URL for HTTP and HTTPS. We use an external proxy, i.e. not one on the local Docker host...it seemed to only affect HTTPS URLs.

What was weird is that it affected not only Docker daemon actions, like docker pull but actions in containers, that have their own proxy configurations and normally wouldn't rely on the docker daemon (such as https_proxy=http://whatev curl -v https://github.com (github being down this morning notwithstanding))

If I can provide any more info I will try but at this time I've rolled back to a previous version. I can re-upgrade cautiously :)

[djs55]

We've fixed a couple of HTTP proxy issues and are hoping to release an update either later today or earlier tomorrow. I'll comment here when the update is available.

Regarding in-container operations that have their own proxy configurations, Docker 17.07 and later have a new CLI feature: docker/cli#93 The file ~/.docker/config.json can contain settings as follows:

{
  "auths" : {

  },
  "orchestrator" : "kubernetes",
  "proxies": {
    "default": {
      "httpProxy": "http://172.16.1.61:3128",
      "httpsProxy": "http://172.16.1.61:3128",
      "noProxy": "",
      "ftpProxy": ""
    }
  }
}

These are automatically inserted by docker run and docker build into the container's environment.

[rubyisbeautiful]

whoa that's big. lots of automation will get shelved now (in a good way) :)
Thanks for fixing this proxy bug quickly @djs55

[djs55]

Sorry I should have emphasised -- we've fixed some proxy bugs for sure, but there might still be one or two left. In particular the original issue reported by @alvarow may not be fixed -- I'm still unable to reproduce it locally (for me using proxies to access a secure registry is working, unfortunately). The update will be worth re-testing, but we might need to release another update soon afterwards.

Thanks for all your patience in the meantime!

[alvarow]

Thanks @djs55, please let me know if there's anything I can do to help. I am happy to demonstrate as well, over a BlueJeans/WebEx/etc session.

[sharenz]

I think I have the same issue. I am behind a proxy with SSL interception. Connecting to external registries (using the proxy) works. But I can not access the internal registry anymore. I think there is a problem with bypassing the proxy for internal domains.

I just downgraded to 17.09.0-ce-mac33 (19543), now it works again.

[danieladams456]

I tried the new 17.12.0-ce-mac47 version and still having several problems. Authentication information is now not passed when set in the proxy settings as http://user:pass@host.domain.net:80.

Also proxy exclusion isn't working in the .domainname.net format (preferred) as well as hostname.domainname.net.

I'm rolling back to 17.09 for now to fix.

(no auth headers passed)

[djs55]

@danieladams456 thanks very much for the report and screenshot. I can reproduce these so I'll work on fixing them and adding some regression tests.