purge_urls() sporadically raises ConnectionError

[berkerpeksag]

I think this was caused by the refactoring we made in f40b171. We now send PURGE requests in the same connection object and for some reason it's being rejected by the server.

Example traceback:

TypeError: getresponse() got an unexpected keyword argument 'buffering'
  File "requests/packages/urllib3/connectionpool.py", line 353, in _make_request
    httplib_response = conn.getresponse(buffering=True)
  File "raven/breadcrumbs.py", line 375, in getresponse
    rv = real_getresponse(self, *args, **kwargs)
ConnectionResetError: [Errno 104] Connection reset by peer
  File "requests/packages/urllib3/connectionpool.py", line 518, in urlopen
    body=body, headers=headers)
  File "requests/packages/urllib3/connectionpool.py", line 355, in _make_request
    httplib_response = conn.getresponse()
  File "raven/breadcrumbs.py", line 375, in getresponse
    rv = real_getresponse(self, *args, **kwargs)
  File "http/client.py", line 1208, in getresponse
    response.begin()
  File "http/client.py", line 380, in begin
    version, status, reason = self._read_status()
  File "http/client.py", line 342, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "python3.4/socket.py", line 374, in readinto
    return self._sock.recv_into(b)
  File "python3.4/ssl.py", line 769, in recv_into
    return self.read(nbytes, buffer)
  File "python3.4/ssl.py", line 641, in read
    v = self._sslobj.read(len, buffer)
ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
  File "requests/adapters.py", line 370, in send
    timeout=timeout
  File "requests/packages/urllib3/connectionpool.py", line 564, in urlopen
    _pool=self, _stacktrace=stacktrace)
  File "requests/packages/urllib3/util/retry.py", line 245, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "requests/packages/urllib3/packages/six.py", line 309, in reraise
    raise value.with_traceback(tb)
  File "requests/packages/urllib3/connectionpool.py", line 518, in urlopen
    body=body, headers=headers)
  File "requests/packages/urllib3/connectionpool.py", line 355, in _make_request
    httplib_response = conn.getresponse()
  File "raven/breadcrumbs.py", line 375, in getresponse
    rv = real_getresponse(self, *args, **kwargs)
  File "http/client.py", line 1208, in getresponse
    response.begin()
  File "http/client.py", line 380, in begin
    version, status, reason = self._read_status()
  File "http/client.py", line 342, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "python3.4/socket.py", line 374, in readinto
    return self._sock.recv_into(b)
  File "python3.4/ssl.py", line 769, in recv_into
    return self.read(nbytes, buffer)
  File "python3.4/ssl.py", line 641, in read
    v = self._sslobj.read(len, buffer)
ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
  File "pythondotorg/manage.py", line 8, in <module>
    execute_from_command_line(sys.argv)
  File "django/core/management/__init__.py", line 371, in execute_from_command_line
    utility.execute()
  File "django/core/management/__init__.py", line 365, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **cmd_options)
  File "django/core/management/base.py", line 335, in execute
    output = self.handle(*args, **options)
  File "peps/management/commands/generate_pep_pages.py", line 51, in handle
    pep0_page, _ = get_pep0_page()
  File "peps/converters.py", line 62, in get_pep0_page
    page.save()
  File "cms/models.py", line 43, in save
    return super().save(**kwargs)
  File "django/db/models/base.py", line 729, in save
    force_update=force_update, update_fields=update_fields)
  File "django/db/models/base.py", line 769, in save_base
    update_fields=update_fields, raw=raw, using=using,
  File "django/dispatch/dispatcher.py", line 178, in send
    for receiver in self._live_receivers(sender)
  File "django/dispatch/dispatcher.py", line 178, in <listcomp>
    for receiver in self._live_receivers(sender)
  File "pages/models.py", line 89, in purge_fastly_cache
    purge_urls(instance.path)
  File "fastly/utils.py", line 20, in purge_urls
    session.request('PURGE', url)
  File "requests/sessions.py", line 461, in request
    resp = self.send(prep, **send_kwargs)
  File "raven/breadcrumbs.py", line 326, in send
    resp = real_send(self, request, *args, **kwargs)
  File "requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)

I've implemented a retry logic in master...berkerpeksag:retry, but I'm not sure if that's the best way to solve this issue. @jaap3 do you have any suggestions?

[berkerpeksag]

OTOH, we are using a horribly outdated version of requests, so we may not use persistent connections at all. I will check requests release notes.

[jaap3]

I don't have the time to look at this in depth. My gut reaction would be to just go back to using a single connection per request and see if that fixes things. I'm not sure why an outdated version of requests is used, so updating that might also be a solution.

…

On Fri, 19 Oct 2018, 08:19 Berker Peksag ***@***.***> wrote: OTOH, we are using a horribly outdated version of requests, so we may not use persistent connections at all. I will check requests release notes. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1349 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC9hcmVCSSWK11P7FnZ4rX8l66Zee_Lks5umW7VgaJpZM4XvqUX> .

[jaap3]

Looking more closely at the stack trace the call to purge_urls is made from the purge_fastly_cache function. In that case it only purges a single URL anyway.

…

On Fri, 19 Oct 2018, 10:16 Jaap Roes ***@***.***> wrote: I don't have the time to look at this in depth. My gut reaction would be to just go back to using a single connection per request and see if that fixes things. I'm not sure why an outdated version of requests is used, so updating that might also be a solution. On Fri, 19 Oct 2018, 08:19 Berker Peksag ***@***.***> wrote: > OTOH, we are using a horribly outdated version of requests, so we may not > use persistent connections at all. I will check requests release notes. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#1349 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAC9hcmVCSSWK11P7FnZ4rX8l66Zee_Lks5umW7VgaJpZM4XvqUX> > . >

[berkerpeksag]

Turns out f40b171 is not the cause of the problem. Here's a recent traceback after I reverted that commit:

ConnectionResetError: [Errno 104] Connection reset by peer
  File "urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "urllib3/connectionpool.py", line 849, in _validate_conn
    conn.connect()
  File "urllib3/connection.py", line 356, in connect
    ssl_context=context)
  File "urllib3/util/ssl_.py", line 359, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "python3.4/ssl.py", line 365, in wrap_socket
    _context=self)
  File "python3.4/ssl.py", line 601, in __init__
    self.do_handshake()
  File "python3.4/ssl.py", line 828, in do_handshake
    self._sslobj.do_handshake()
ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
  File "requests/adapters.py", line 445, in send
    timeout=timeout
  File "urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "urllib3/util/retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "urllib3/connectionpool.py", line 849, in _validate_conn
    conn.connect()
  File "urllib3/connection.py", line 356, in connect
    ssl_context=context)
  File "urllib3/util/ssl_.py", line 359, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "python3.4/ssl.py", line 365, in wrap_socket
    _context=self)
  File "python3.4/ssl.py", line 601, in __init__
    self.do_handshake()
  File "python3.4/ssl.py", line 828, in do_handshake
    self._sslobj.do_handshake()
ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
  File "pythondotorg/manage.py", line 8, in <module>
    execute_from_command_line(sys.argv)
  File "django/core/management/__init__.py", line 371, in execute_from_command_line
    utility.execute()
  File "django/core/management/__init__.py", line 365, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **cmd_options)
  File "django/core/management/base.py", line 335, in execute
    output = self.handle(*args, **options)
  File "peps/management/commands/generate_pep_pages.py", line 79, in handle
    p = get_pep_page(pep_number)
  File "peps/converters.py", line 196, in get_pep_page
    pep_page.save()
  File "cms/models.py", line 43, in save
    return super().save(**kwargs)
  File "django/db/models/base.py", line 729, in save
    force_update=force_update, update_fields=update_fields)
  File "django/db/models/base.py", line 769, in save_base
    update_fields=update_fields, raw=raw, using=using,
  File "django/dispatch/dispatcher.py", line 178, in send
    for receiver in self._live_receivers(sender)
  File "django/dispatch/dispatcher.py", line 178, in <listcomp>
    for receiver in self._live_receivers(sender)
  File "pages/models.py", line 89, in purge_fastly_cache
    purge_url('/{}'.format(instance.path))
  File "fastly/utils.py", line 18, in purge_url
    headers={'Fastly-Key': api_key},
  File "requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "raven/breadcrumbs.py", line 326, in send
    resp = real_send(self, request, *args, **kwargs)
  File "requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "requests/adapters.py", line 495, in send
    raise ConnectionError(err, request=request)

[jaap3]

@berkerpeksag have you figured this out yet? Did something change on the server? e.g. Python version, OpenSSL. Or did Fastly change something on their end related to SSL/TLS? All the ConnectionErrors seem to be related to SSL handling

[berkerpeksag]

No, I don't have time to investigate it further the next two weeks. I don't think we've changed anything on the server.