WebSocket fails with server on Google Cloud Run

[buehren]

Problem you have encountered:

We are currently evaluating Google Cloud Run and Flutter. The project requires a WebSocket connection from a Flutter App to a server container running in Google Cloud Run.

But the combination of a Dart client + Cloud Run seems to be incompatible regarding WebSockets:

• The WebSocket connection is established, and the Flutter/Dart client receives messages sent by the server on Cloud Run.

• As soon as the Dart client sends a message to the server, the connection is dropped.

  It happens with both a Java/Quarkus and a Python server in Cloud Run.
  It does not happen with those servers on localhost in Docker.

  It happens both in a Flutter App running on an Android device and in a Dart program executed from the command line.
  It does not happen with a Python client connecting the same servers.

So the only combination that fails is the one that we need: Dart + Cloud Run.

What you expected to happen:

I expect the WebSocket connection to remain intact if the Dart client sends messages over it.

Steps to reproduce:

I have created a complete bug reproducer with clients in Dart and Python and servers in Java and Python:

• Dart client (the README contains the outputs of the messages sent and received and the aborted connection):
  https://github.com/buehren/websocket-client-dart

• Java/Quarkus WebSocket Echo Server (including a "Run in Google Cloud" button):
  https://github.com/buehren/websocket-echo-server-quarkus

• Python WebSocket Echo Server (including a "Run in Google Cloud" button):
  https://github.com/buehren/websocket-echo-server-python

Other information (workarounds you have tried, documentation consulted, etc):

I could not find helpful information in the logs of Google Cloud Run.

I could not find any hint by debugging the Dart client.

• Dart SDK Version (dart --version)
  Dart SDK version: 2.10.5 (stable) (Unknown timestamp) on "linux_x64"
  Dart SDK version: 2.10.5 (stable) (Tue Jan 19 13:05:37 2021 +0100) on "linux_x64"
  Dart SDK version: 2.12.0-259.12.beta (beta) (Fri Feb 12 11:00:34 2021 +0100) on "windows_x64"

• Whether you are using Windows, MacOSX, or Linux (if applicable)
  Windows, Linux, Android

• Whether you are using Chrome, Safari, Firefox, Edge (if applicable)
  not applicable

[buehren]

Some code to clarify the problem (taken from my reproducer example):

void run(String webSocketUrl) async {
  ...
  Future<WebSocket> futureWebSocket = WebSocket.connect(webSocketUrl);
  futureWebSocket.then((WebSocket ws) {
    _webSocket = ws;
    print(new DateTime.now().toString() + " WebSocket readyState: " + (_webSocket.readyState.toString()));

    _webSocket.listen((data) {
      print(new DateTime.now().toString() + " Received data: $data");
    }, onError: _error, onDone: _done);

    // send messages

    new Timer(const Duration(seconds: 1), () {
      print(new DateTime.now().toString() + " Sending 'hello'");
      _webSocket.add('hello');
    });
    ...
}

_done() async {
  print(new DateTime.now().toString() + " CONNECTION DONE! \n" +
      "readyState=" + _webSocket.readyState.toString() + "\n" +
      "closeCode=" + _webSocket.closeCode.toString() + "\n" +
      "closeReason=" + _webSocket.closeReason.toString() + "\n");
}

The Dart client works with WebSocket echo server running on localhost (or with ws://echo.websocket.org):

> dart run wsclient.dart --url ws://localhost:8080/websocket

2021-02-26 21:25:22.293451 Starting connection attempt to ws://localhost:8080/websocket ...
2021-02-26 21:25:22.418794 WebSocket readyState: 1
2021-02-26 21:25:23.446877 Sending 'hello'
2021-02-26 21:25:23.458083 Received data: echo of hello
2021-02-26 21:25:24.422013 Sending 'how are you?'
2021-02-26 21:25:24.422752 Received data: echo of how are you
2021-02-26 21:25:26.433450 Sending 'still there?'
2021-02-26 21:25:26.436280 Received data: echo of still there?
^C
^C

The Dart client does NOT work with WebSocket echo server running on Google Cloud Run — the WebSocket is closed right after the Dart client sends data:

> dart run wsclient.dart --url wss://websocket-echo-server-quarkus-xxxxxxxxxx-xx.x.run.app/websocket

2021-02-26 21:25:40.539316 Starting connection attempt to wss://websocket-echo-server-quarkus-xxxxxxxxxx-xx.x.run.app/websocket ...
2021-02-26 21:25:40.844298 WebSocket readyState: 1
2021-02-26 21:25:41.851590 Sending 'hello'
2021-02-26 21:25:41.898587 CONNECTION DONE!
readyState=3
closeCode=1005
closeReason=

2021-02-26 21:25:42.847345 Sending 'how are you?'
2021-02-26 21:25:44.848959 Sending 'still there?'

[mit-mit]

cc @kevmoo @subfuzion, any idea what's going on here?

[buehren]

Hi @mit-mit @kevmoo @subfuzion @srawlins,

thanks for looking into this.

Meanwhile I have created a reproducer Flutter App and tested sending messages to the server on Cloud Run over the WebSocket on various target platforms:

✗ Flutter for Android
✗ Flutter for Linux
✗ Flutter for Windows
✓ Flutter for Web

As WebSocket from dart:io does not work in Flutter for Web ("Error: Unsupported operation: Platform._version"), I used WebSocketChannel instead (from package:web_socket_channel). WebSocketChannel uses WebSocket from dart:html for the Web target and from dart:io for the other targets (see html.dart and io.dart).

So it seems that the problem must be located in WebSocket from dart:io (or in any of its dependencies).

To easily reproduce the issue with dart:io:

1. Start Java (Quarkus) server on Google Cloud Run (to prove that it is not a problem of the Java implementation you can also start a Python server instead).

2. Run the reproducer Dart command line tool:
   dart run wsclient.dart --url wss://websocket-echo-server-quarkus-xxxxxxxxxx-xx.x.run.app/websocket
   (replace xxxxxxxxxx-xx.x with the corresponding part of the Java server's URL that was displayed after starting the server on Cloud Run).

Thank you very much

buehren

[kevmoo]

Seems like this is more to do with the the web_socket CLIENT impl in dart:io.

CC @a-siva

[a-siva]

/cc @aam

[buehren]

Exactly, Dart is used in the client in this scenario. Thank you very much for looking into this.

[brianschardt]

use it is the content-length flutter and dart:io is automatically seting it. It should not do this.
i found a fix
request.headers.removeAll('content-length');

Future<WebSocket> createWebsocket(
  String url, {
  Iterable<String> protocols,
  Map<String, dynamic> headers,
  Duration pingInterval,
}) async {
  print("CREATING WEBSOCKET");
  final HttpClient _httpClient = HttpClient();
  Uri uri = Uri.parse(url);

  final Random random = Random();
  // Generate 16 random bytes.
  final Uint8List nonceData = Uint8List(16);
  for (int i = 0; i < 16; i++) {
    nonceData[i] = random.nextInt(256);
  }
  final String nonce = base64.encode(nonceData);
  uri = Uri(
      scheme: uri.scheme == "wss" ? "https" : "http",
      userInfo: uri.userInfo,
      host: uri.host,
      port: uri.port,
      path: uri.path,
      query: uri.query,
      fragment: uri.fragment);
  final HttpClientRequest request = await _httpClient.openUrl('GET', uri);
  if (uri.userInfo != null && !uri.userInfo.isEmpty) {
    print(uri.userInfo);
    // If the URL contains user information use that for basic
    // authorization.
    final String auth = base64.encode(utf8.encode(uri.userInfo));
    request.headers.set(HttpHeaders.authorizationHeader, 'Basic $auth');
  }
  // Setup the initial handshake.
  request.headers
    ..set(HttpHeaders.connectionHeader, 'Upgrade')
    ..set(HttpHeaders.upgradeHeader, 'websocket')
    ..set('Sec-WebSocket-Key', nonce)
    ..set('Cache-Control', 'no-cache')
    ..set('Sec-WebSocket-Version', '13')
    ..set('Sec-WebSocket-Protocol', protocols.first);
  request.headers.removeAll('content-length');
  if (headers != null) {
    headers.forEach((key, value) {
      print('setting header ${HttpHeaders.authorizationHeader} ${value}');
      request.headers.set(HttpHeaders.authorizationHeader, value);
    });
  }

  print(request.headers);
  HttpClientResponse response = await request.close();

  final Socket socket = await response.detachSocket();
  final WebSocket websocket = WebSocket.fromUpgradedSocket(socket,
      protocol: protocols.first, serverSide: false);
  websocket.pingInterval = pingInterval;

  return websocket;
}

But now with some package updates to flutter 2 my fixes no longer work because if Websocket Channels.
Can the flutter team allow us to change or modify headers in websocket creation?

[aam]

@brianschardt what was the last dart/flutter version your fix with removing 'content-length' for dart webservice client to connect with Google Cloud Run worked on?

[thomas-ferchau]

Hi @aam @a-siva @kevmoo

Removing the content-length header would fix the problem for me, too. (Thank you, @brianschardt!)

I tried that by copying the _http library of the Dart SDK and adding this to _WebSocketImpl.connect(...) in websocket_impl.dart:

      // Remove header "content-length". It is automatically set to 0 for GET requests.
      // But there are servers that terminate the connection when sending
      // data even after the Websocket upgrade (e.g. Google Cloud Run).
      // https://github.com/dart-lang/sdk/issues/45139
      request.headers.removeAll(HttpHeaders.contentLengthHeader);

You can also see it in this commit:
buehren/websocket-client-dart@404e875

With this modification, the WebSocket client works with Google Cloud Run. I have prepared a branch containing the change. You can try it like this:

git clone https://github.com/buehren/websocket-client-dart
cd websocket-client-dart/
git checkout fix-websocket # this branch contains the fix

dart pub get

# Still works with echo.websocket.org:
dart run wsclient.dart 

# Now works with server at Google Cloud Run (my Java/Quarkus echo server in this case):
dart run wsclient.dart --url wss://websocket-echo-server-quarkus-xxxxxxxxxx-xx.x.run.app/websocket

(You can easily launch that WebSocket server on Google Cloud Run here: https://github.com/buehren/websocket-echo-server-quarkus)

The content-length header is automatically set to 0 for GET requests. But this is not true for WebSocket connections because you can send data after the upgrade.

I suggest removing the content-length header in _WebSocketImpl.connect(...).
If that is not possible for some reason, there should be an option to do so.

Other client libraries do not send the header either, e.g. this Python library:
https://github.com/websocket-client/websocket-client

Connecting with the Python websocket-client library....

python /usr/local/bin/wsdump.py "wss://websocket-echo-server-python-xxxxxxxxxx-xx.x.run.app" -n

...does not produce the request header content-length: 0 (and works):

> mitmdump  -vv
Proxy server listening at http://*:8080

127.0.0.1:36342: clientconnect
::ffff:127.0.0.1:36342: Set new server address: websocket-echo-server-python-xxxxxxxxxx-xx.x.run.app:443
::ffff:127.0.0.1:36342: Establish TLS with client
::ffff:127.0.0.1:36342: request
  -> Request(GET /)
::ffff:127.0.0.1:36342: serverconnect
  -> ('websocket-echo-server-python-xxxxxxxxxx-xx.x.run.app', 443)
::ffff:127.0.0.1:36342: Establish TLS with server
::ffff:127.0.0.1:36342: ALPN selected by server: -
::ffff:127.0.0.1:36342: response
  -> Response(101, no content)
127.0.0.1:36342: GET https://websocket-echo-server-python-xxxxxxxxxx-xx.x.run.app/
    Upgrade: websocket
    Host: websocket-echo-server-python-xxxxxxxxxx-xx.x.run.app
    Origin: http://websocket-echo-server-python-xxxxxxxxxx-xx.x.run.app
    Sec-WebSocket-Key: pVcdIQOu3JlvNl7O43ynWQ==
    Sec-WebSocket-Version: 13
    Connection: Upgrade
 << 101 Switching Protocols 0b
    sec-websocket-accept: 0UIkeXewbQ85Q4zvC0xC3B+pePQ=
    date: Tue, 27 Apr 2021 20:14:33 GMT
    server: Google Frontend
    Upgrade: websocket
    Connection: Upgrade
    Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
127.0.0.1:36342 -> WebSocket 1 message -> websocket-echo-server-python-xxxxxxxxxx-xx.x.run.app:443/
127.0.0.1:36342 <- WebSocket 1 message <- websocket-echo-server-python-xxxxxxxxxx-xx.x.run.app:443/

...

Connecting with Dart's WebSocket client makes Google Cloud Run terminate the connection when sending data from the client...

> /snap/flutter/current/dart.sh run wsclient.dart --url ws://websocket-echo-server-python-xxxxxxxxxx-xx.x.run.app/

2021-04-27 20:14:59.545995 Starting connection attempt to ws://websocket-echo-server-python-xxxxxxxxxx-xx.x.run.app/ ...
2021-04-27 20:15:00.209132 WebSocket readyState: 1
2021-04-27 20:15:01.223964 Sending 'hello'
2021-04-27 20:15:01.382507 CONNECTION DONE!
readyState=3
closeCode=1005
closeReason=

2021-04-27 20:15:02.217821 Sending 'how are you?'
2021-04-27 20:15:04.214290 Sending 'still there?'

...because of the request header content-length: 0:

> mitmdump  -vv
Proxy server listening at http://*:8080

127.0.0.1:36424: clientconnect
::ffff:127.0.0.1:36424: Set new server address: websocket-echo-server-python-xxxxxxxxxx-xx.x.run.app:443
::ffff:127.0.0.1:36424: Establish TLS with client
::ffff:127.0.0.1:36424: request
  -> Request(GET /)
::ffff:127.0.0.1:36424: serverconnect
  -> ('websocket-echo-server-python-xxxxxxxxxx-xx.x.run.app', 443)
::ffff:127.0.0.1:36424: Establish TLS with server
::ffff:127.0.0.1:36424: ALPN selected by server: -
::ffff:127.0.0.1:36424: response
  -> Response(101, no content)
127.0.0.1:36424: GET https://websocket-echo-server-python-xxxxxxxxxx-xx.x.run.app/
    user-agent: Dart/2.12 (dart:io)
    connection: Upgrade
    cache-control: no-cache
    accept-encoding: gzip
    content-length: 0
    sec-websocket-version: 13
    host: websocket-echo-server-python-xxxxxxxxxx-xx.x.run.app
    sec-websocket-key: kURzWIIFk0KyZaOjhGwIDQ==
    upgrade: websocket
 << 101 Switching Protocols 0b
    sec-websocket-accept: F2MeBPvouzDEWo1HWVS4A7OFCVI=
    date: Tue, 27 Apr 2021 20:19:19 GMT
    server: Google Frontend
    Upgrade: websocket
    Connection: Upgrade
    Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
127.0.0.1:36424 -> WebSocket 2 message -> websocket-echo-server-python-xxxxxxxxxx-xx.x.run.app:443/
Error in WebSocket connection to websocket-echo-server-python-xxxxxxxxxx-xx.x.run.app:443: WebSocket connection closed unexpectedly by server: TcpDisconnect(None,)
WebSocket connection closed by client: 1000 (message missing), unknown status code
::ffff:127.0.0.1:36424: serverdisconnect
  -> ('websocket-echo-server-python-xxxxxxxxxx-xx.x.run.app', 443)
127.0.0.1:36424: clientdisconnect

[aam]

thanks @thomas-ferchau .
I believe we should drop content-length:0 from all GET or HEAD requests, not only websocket calls.
This is based on what https://tools.ietf.org/html/rfc7230#section-3.3.2 says
"A user agent SHOULD NOT send a
Content-Length header field when the request message does not contain
a payload body and the method semantics do not anticipate such a
body."