Expand proxy support: Authenticated Proxy and Websockets (`HTTP CONNECT` tunnel)

[mattheworiordan]

A customer has requested the ability to set the proxy manually for HTTP and WebSocket connections.

We need to add this functionality to the library.

┆Issue is synchronized with this Jira Task by Unito

[paddybyers]

A few requirements questions:

• for HTTP proxy, I assume we want support for an explicit proxy, which supports HTTP, or HTTPS as a MITM proxy;
• do we need to support configuration of proxy host and port explicitly in the client API which targets only the Ably host, or should we rely on the JVM-wide system properties? If the latter, do we populate those properties based on given client options, or is that configured outside our control?
• do we need to support proxy authentication? If so, the same questions apply as above: will we use the JVM-wide properties, or enable it only for proxied connections to the Ably host?
• is there any requirement to support tunnelling?

For websockets, would the aim be to support explicit proxying in the same way?

If implementing comet as a fallback to websockets, assuming that the proxy cannot support websockets, will we be able to support streamed comet or must it be polled? If we support both, do we have to detect which will work, or can we rely on a client configuration?

[mattheworiordan]

@paddybyers I will ask the customer now to see if he can comment to help direct us.

[Frintrop]

• yes. explicit proxy ( java.net.Proxy object?)
  -not system wide proxy. some companies have more than one proxy. setter i.e. as java.net.Proxy object ) to connect method.
  proxy authentication would be nice, too.
  -for websockets, too. 50 percent of proxies don´t support websocket, 50 percent works. seamly fallback to comet first and than polled would be fine. rely on a client configuration may be possible as we have a b2b applicaiton and not c2b..

Might this helpful?

[Frintrop]

Think wasync (configured with async http client ) from atmosphere framkework supports this setup ...

[paddybyers]

explicit proxy ( java.net.Proxy object?) -not system wide proxy

Yes, agreed, this should be used, configured by proxy options given when initialising the Ably library.

proxy authentication would be nice, too

This seems to be slightly more problematic using HttpURLConnection because the way to do this seems to be with Authenticator.setDefault(), which sets a global handler, not one for a specific context. We need to look into this further.

for websockets, too. 50 percent of proxies don´t support websocket, 50 percent works

OK, we will look into this.

fallback to comet first and than polled would be fine. rely on a client configuration may be possible as we have a b2b applicaiton and not c2b..

OK. We would prefer to make it fallback seamlessly, but an interim solution may be achievable more quickly by having an explicit transports = new String[]{ "comet" } so we avoid the complications of implementing the fallback sequence in the first instance.

[mattheworiordan]

@gokhanbarisaker I have added Proxy support to Ruby, but unfortunately it's blocked by a WebSocket issue so I've only been able to test the HTTP parts. See ably/ably-ruby#84

We would like to do the same for the Java library, see @paddybyers branch at https://github.com/ably/ably-java/tree/proxy-support. Can you add suitable test coverage to his work, and:

• add support for java.net.Proxy, initialised using host/port configured with new members in ClientOptions. The exact options and naming we'll review in your PR so don't worry too much about what you choose.
• Proxy settings will apply to all HTTP requests
• Please ensure you have tests against sandbox-proxy.ably.io
• Once done, we need to look further into how to enable proxy authentication without it being a global setting

Later we'll then need to address how this applies to WebSockets, and #121 will address fallback Comet support.

[paddybyers]

We would like to do the same for the Java library

See the latest commits to that branch.

add support for java.net.Proxy, initialised using host/port configured with new members in ClientOptions. The exact options and naming we'll review in your PR so don't worry too much about what you choose.

This is done.

Proxy settings will apply to all HTTP requests

I've added ProxyOptions.nonProxyHosts as an array of regexp strings for hostnames that are not to be proxied.

Please ensure you have tests against sandbox-proxy.ably.io

I think this is the first thing to look at: get some tests running against that (without authentication initially); we should decide what constitutes appropriate coverage. There should be a http and async http request which successfully uses a configured proxy, and another pair of requests that bypass the proxy based on a nonProxyHosts configuration. We need to work out how we would know that the proxy was or was not used.

we need to look further into how to enable proxy authentication without it being a global setting

Having thought about this a bit more this should be simple enough, but we'll wait until the proxy is enabled for authentication to try things out.

[mattheworiordan]

We need to work out how we would know that the proxy was or was not used.

Easy enough, see ably/ably-ruby@cf43713#diff-d4400109d5e5b080196aad253c2e7f4fR39 and ably/ably-ruby@cf43713#diff-d4400109d5e5b080196aad253c2e7f4fR72. It checks that the proxy fails when the details are incorrect or TLS was supposed to enabled etc.

[paddybyers]

@gokhanbarisaker: please see 67b5961

I've added auth support and a few initial tests to run against sandbox-proxy.ably.io.

[paddybyers]

Reopening after Github auto-closed.

The current shortfalls are:

• no support for auth when using TLS. This is because although we set the Proxy-Authorization header on a connection following a 407, it is not added to the CONNECT request. We can set a global Authenticator which might fix the issue, but that would be a temporary fix because it's not really acceptable for us to set a global Authenticator. I can't see a way of us setting a per-connection Authenticator using the HttpURLConnection API so ultimately we would need to go another route, either to use an alternative library, eg okhttp
• no support for websockets. This is because the websocket library we currently use does not support CONNECT and tunnelling. (It does support a SOCKS proxy). To address this we would either need to add support to that library, or switch to an alternative websocket library. okhttp also supports ws.
• digest auth not working. It is implemented but so far not passing tests.

I think we've nearly gone as far as we can with this without looking again at the http API we use. I do think as a short term improvement we should set a global authenticator if proxy auth is configured, and probably add some more tests.