Skip to content

Address requests/urllib3 errors arising from OpenTelemetry API use #4839

Open
@StevenMaude

Description

These errors occur frequently, and were originally related to an upgrade of urllib3, but downgrading urllib3 no longer resolves the errors (and wouldn't be an ideal long-term solution anyway as urllib3 v1 is not really actively developed).

We're on the latest OpenTelemetry package versions, and this still occurs.

OpenCodelists also uses OpenTelemetry, but with a simpler setup, and doesn't see similar errors, so it's possibly something particular to how we're using OpenTelemetry in job-server, or perhaps some difference in deployment.

It might be a little tricky to reproduce and debug this because:

  • this occurs when exporting spans to Honeycomb, and involves their API, though I believe we can get a test API key
  • it may also involve the server stack in deployment, which we don't run in its entirety for local development

It's probably worth trying to reproduce errors with Django alone running locally, using a test API key, and if that doesn't work, thinking about whether other bits of the stack are needed to be also run locally to reproduce it.

Example error from #4715 (comment):

Message

Exception while exporting Span batch.

Stack Trace

RemoteDisconnected: Remote end closed connection without response
  File "urllib3/connectionpool.py", line 716, in urlopen
    httplib_response = self._make_request(
  File "urllib3/connectionpool.py", line 468, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
    # Permission is hereby granted, free of charge, to any person obtaining a copy
  File "urllib3/connectionpool.py", line 463, in _make_request
    httplib_response = conn.getresponse()
  File "http/client.py", line 1428, in getresponse
    response.begin()
  File "http/client.py", line 331, in begin
    version, status, reason = self._read_status()
  File "http/client.py", line 300, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"

ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
  File "requests/adapters.py", line 667, in send
    resp = conn.urlopen(
  File "urllib3/connectionpool.py", line 802, in urlopen
    retries = retries.increment(
  File "urllib3/util/retry.py", line 552, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "urllib3/connectionpool.py", line 716, in urlopen
    httplib_response = self._make_request(
  File "urllib3/connectionpool.py", line 468, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
    # Permission is hereby granted, free of charge, to any person obtaining a copy
  File "urllib3/connectionpool.py", line 463, in _make_request
    httplib_response = conn.getresponse()
  File "http/client.py", line 1428, in getresponse
    response.begin()
  File "http/client.py", line 331, in begin
    version, status, reason = self._read_status()
  File "http/client.py", line 300, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"

ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
  File "/opt/venv/lib/python3.12/site-packages/opentelemetry/sdk/trace/export/__init__.py", line 360, in _export_batch
    self.span_exporter.export(self.spans_list[:idx])  # type: ignore
  File "/opt/venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 189, in export
    return self._export_serialized_spans(serialized_data)
  File "/opt/venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 159, in _export_serialized_spans
    resp = self._export(serialized_data)
  File "/opt/venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 133, in _export
    return self._session.post(
  File "requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/venv/lib/python3.12/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 180, in instrumented_send
    return wrapped_send(self, request, **kwargs)
  File "requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "requests/adapters.py", line 682, in send
    raise ConnectionError(err, request=request)

Metadata

Assignees

No one assigned

    Labels

    deck-scrubbingTech debt or other between-initiative tidy-up work

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions