Connection reset by peer in logging client libs

[devashishshankar]

I use gcloud logging and datastore client libs on my GCE VMs. Sometimes (rather sporadically) my jobs fail due to connection reset by peer. The only common pattern is that it happens in long running jobs.

The stacktrace goes something like this:

  File "/usr/local/lib/python2.7/dist-packages/gcloud/logging/logger.py", line 189, in log_text
    client.logging_api.write_entries([entry_resource])
  File "/usr/local/lib/python2.7/dist-packages/gcloud/logging/connection.py", line 150, in write_entries
    data=data)
  File "/usr/local/lib/python2.7/dist-packages/gcloud/connection.py", line 343, in api_request
    target_object=_target_object)
  File "/usr/local/lib/python2.7/dist-packages/gcloud/connection.py", line 241, in _make_request
    return self._do_request(method, url, headers, data, target_object)
  File "/usr/local/lib/python2.7/dist-packages/gcloud/connection.py", line 270, in _do_request
    body=data)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/transport.py", line 169, in new_request
    redirections, connection_type)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1609, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1351, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1307, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1073, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 415, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 371, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/ssl.py", line 714, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 608, in read
    v = self._sslobj.read(len or 1024)
error: [Errno 104] Connection reset by peer

I'm not totally sure, but this could be because of the connection pool being maintained by the underlying httplib2 library. httplib2 is holding connection pools, but doesn't seem to be refreshing them. So, it could be that it holds a connection which has become stale (if no request happened in a while), and thus server sends a connection reset.

I suspect some of my datastore requests are also failing due to a similar issue.

[dhermes]

Closing as duplicate of #2028