Description
These errors occur frequently, and were originally related to an upgrade of urllib3
, but downgrading urllib3
no longer resolves the errors (and wouldn't be an ideal long-term solution anyway as urllib3
v1 is not really actively developed).
We're on the latest OpenTelemetry package versions, and this still occurs.
OpenCodelists also uses OpenTelemetry, but with a simpler setup, and doesn't see similar errors, so it's possibly something particular to how we're using OpenTelemetry in job-server, or perhaps some difference in deployment.
It might be a little tricky to reproduce and debug this because:
- this occurs when exporting spans to Honeycomb, and involves their API, though I believe we can get a test API key
- it may also involve the server stack in deployment, which we don't run in its entirety for local development
It's probably worth trying to reproduce errors with Django alone running locally, using a test API key, and if that doesn't work, thinking about whether other bits of the stack are needed to be also run locally to reproduce it.
Example error from #4715 (comment):
Message
Exception while exporting Span batch.
Stack Trace
RemoteDisconnected: Remote end closed connection without response
File "urllib3/connectionpool.py", line 716, in urlopen
httplib_response = self._make_request(
File "urllib3/connectionpool.py", line 468, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
# Permission is hereby granted, free of charge, to any person obtaining a copy
File "urllib3/connectionpool.py", line 463, in _make_request
httplib_response = conn.getresponse()
File "http/client.py", line 1428, in getresponse
response.begin()
File "http/client.py", line 331, in begin
version, status, reason = self._read_status()
File "http/client.py", line 300, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
File "requests/adapters.py", line 667, in send
resp = conn.urlopen(
File "urllib3/connectionpool.py", line 802, in urlopen
retries = retries.increment(
File "urllib3/util/retry.py", line 552, in increment
raise six.reraise(type(error), error, _stacktrace)
File "urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "urllib3/connectionpool.py", line 716, in urlopen
httplib_response = self._make_request(
File "urllib3/connectionpool.py", line 468, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
# Permission is hereby granted, free of charge, to any person obtaining a copy
File "urllib3/connectionpool.py", line 463, in _make_request
httplib_response = conn.getresponse()
File "http/client.py", line 1428, in getresponse
response.begin()
File "http/client.py", line 331, in begin
version, status, reason = self._read_status()
File "http/client.py", line 300, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
File "/opt/venv/lib/python3.12/site-packages/opentelemetry/sdk/trace/export/__init__.py", line 360, in _export_batch
self.span_exporter.export(self.spans_list[:idx]) # type: ignore
File "/opt/venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 189, in export
return self._export_serialized_spans(serialized_data)
File "/opt/venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 159, in _export_serialized_spans
resp = self._export(serialized_data)
File "/opt/venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 133, in _export
return self._session.post(
File "requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
File "requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/opt/venv/lib/python3.12/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 180, in instrumented_send
return wrapped_send(self, request, **kwargs)
File "requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "requests/adapters.py", line 682, in send
raise ConnectionError(err, request=request)