net/http: Server rejects CONNECT requests without a Host header, per the spec

[johnmah]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.7.4 (problem exists on master branch as well)

What operating system and processor architecture are you using (go env)?

linux-amd64

What did you do?

Sample code here: https://play.golang.org/p/dpLc5rnHJZ

We have created a proxy server using the net/http package. If a HTTP client using the proxy server issues a CONNECT request without a Host: header, the golang HTTP server implementation will close the connection with 400 Bad Request: missing required Host header error.

CONNECT api.surfeasy.com:443 HTTP/1.1
...

as opposed to:

CONNECT api.surfeasy.com:443 HTTP/1.1
Host: api.surfeasy.com:443
...

Now this could be a contentious issue as the HTTP 1.1 spec (see https://tools.ietf.org/html/rfc7230#section-5.4) specifies that a Host header is mandatory, but we see some VERY popular devices (i.e.: iOS devices) implement proxy clients that do not send the Host: header when configured to use proxies (and as a result issue CONNECT requests). This could be an artifact of interop with older HTTP 1.0 proxies.

It appears that the HTTP request code in src/net/http/request.go will synthesize a Host attribute based on the CONNECT parameters (authority-form) and ignores any Host: header, so relaxing the Host header restriction would allow for better compatibility. As it stands now, any HTTP server implementing some form of CONNECT handler might not work with a large range of devices.

What did you expect to see?

Client CONNECT request completes as expected and leaves connection open for further requests.

What did you see instead?

net/http Server implementation returns a 400 Bad Request: missing required Host header and closes the connection.

[bradfitz]

Does iOS itself have this bug, or certain app(s)?

Did an older version of various specs permit CONNECT without a host header?

I'm inclined to say that you can use http.ReadRequest and handle this yourself if you're a proxy supporting buggy clients, unless you can convince me that this is widespread.

/cc @tombergan

[johnmah]

I am going to need some more time to look at some proper tcpdumps but we see this issue with iOS (set to use a HTTP proxy written in Go) and the native iOS Facebook app. I have heard rumours that Facebook may bypass the default iOS networking stack but I don't have any concrete evidence.

The original proxy spec written by Ari Luotonen in 1998 did not require a Host header. (see https://tools.ietf.org/html/draft-luotonen-web-proxy-tunneling-01)

I do have a branch that I can submit with a sample 1-line fix.

[bradfitz]

draft-luotonen-web-proxy-tunneling-01 shows examples using HTTP/1.0, though.

[johnmah]

I understand that Ari's draft uses HTTP 1.0 but the spec for CONNECT (https://tools.ietf.org/html/rfc2817, 2000) was only superseded by the HTTP 1.1 (https://tools.ietf.org/html/rfc7230, 2014). At the time of RFC2817 CONNECT was a reserved method but not official.

My fix looks like this:

diff --git a/src/net/http/server.go b/src/net/http/server.go
index 6df9c260e4..c0b8914ee1 100644
--- a/src/net/http/server.go
+++ b/src/net/http/server.go
@@ -936,7 +936,7 @@ func (c *conn) readRequest(ctx context.Context) (w *response, err error) {
 
        hosts, haveHost := req.Header["Host"]
        isH2Upgrade := req.isH2Upgrade()
-       if req.ProtoAtLeast(1, 1) && (!haveHost || len(hosts) == 0) && !isH2Upgrade {
+       if req.ProtoAtLeast(1, 1) && ((!haveHost || len(hosts) == 0) && len(req.Host) == 0) && !isH2Upgrade {
                return nil, badRequestError("missing required Host header")
        }
        if len(hosts) > 1 {

[bradfitz]

Yes, the change itself is trivial. (adding a test will likely be much more code)

This bug isn't about figuring out the change, though, but about making a decision on whether we should, understanding how wide-spread this is or if it was ever allowed.

If you want a change to go in, convince me of either:

• this is widespread
• an RFC used to allow this (HTTP/1.1 + CONNECT with no host header)

Related: #14952 is about adding opt-in knobs for accepting malformed HTTP.

[johnmah]

Understood. We will have more data in the upcoming weeks as we start more in-depth testing across the entire app ecosystem. Breaking the native Facebook app is a doozy though.

[bradfitz]

Facebook might be big enough on its own to make this worth changing.

Let's still figure out whose networking stack it is first. Then, if Facebook, we can ask Facebook to fix it, or at least see what they say. Maybe they'll just fix it quickly.

If it's Apple, that's a bit harder, and almost certainly worth adding a special case at that point.

[johnmah]

I can take care of gathering the data for this issue in the next few days and will update here.

[bradfitz]

@johnmah, any update?