Repository navigation
Investigate/test fetcher retries #2842
Copy link
Copy link
Open
Description
I'm expecting urllib3 to retry 3 times on common http failures when using urllib3fetcher
There's an interesting failure in sigstore-probers where this does not seem to happen:
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/urllib3/response.py", line 754, in _error_catcher
yield
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/urllib3/response.py", line 879, in _raw_read
data = self._fp_read(amt, read1=read1) if not fp_closed else b""
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/urllib3/response.py", line 862, in _fp_read
return self._fp.read(amt) if amt is not None else self._fp.read()
~~~~~~~~~~~~~^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/http/client.py", line 479, in read
s = self.fp.read(amt)
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/socket.py", line 719, in readinto
return self._sock.recv_into(b)
~~~~~~~~~~~~~~~~~~~~^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/ssl.py", line 1304, in recv_into
return self.read(nbytes, buffer)
~~~~~~~~~^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/ssl.py", line 1138, in read
return self._sslobj.read(len, buffer)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
TimeoutError: The read operation timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.13.4/x64/bin/tuf-on-ci-test-client", line 8, in <module>
sys.exit(client())
~~~~~~^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/click/core.py", line 1161, in __call__
return self.main(*args, **kwargs)
~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/click/core.py", line 1082, in main
rv = self.invoke(ctx)
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/click/core.py", line 1443, in invoke
return ctx.invoke(self.callback, **ctx.params)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/click/core.py", line 788, in invoke
return __callback(*args, **kwargs)
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/tuf_on_ci/client.py", line 84, in client
updater.refresh()
~~~~~~~~~~~~~~~^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/tuf/ngclient/updater.py", line 170, in refresh
self._load_timestamp()
~~~~~~~~~~~~~~~~~~~~^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/tuf/ngclient/updater.py", line 425, in _load_timestamp
data = self._download_metadata(
Timestamp.type, self.config.timestamp_max_length
)
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/tuf/ngclient/updater.py", line 313, in _download_metadata
return self._fetcher.download_bytes(url, length)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/tuf/ngclient/fetcher.py", line 139, in download_bytes
with self.download_file(url, max_length) as dl_file:
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/contextlib.py", line 141, in __enter__
return next(self.gen)
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/tuf/ngclient/fetcher.py", line 102, in download_file
for chunk in chunks:
^^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/tuf/ngclient/urllib3_fetcher.py", line 105, in _chunks
yield from response.stream(self.chunk_size)
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/urllib3/response.py", line 1066, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/urllib3/response.py", line 955, in read
data = self._raw_read(amt)
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/urllib3/response.py", line 878, in _raw_read
with self._error_catcher():
~~~~~~~~~~~~~~~~~~~^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/contextlib.py", line 162, in __exit__
self.gen.throw(value)
~~~~~~~~~~~~~~^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/urllib3/response.py", line 759, in _error_catcher
raise ReadTimeoutError(self._pool, None, "Read timed out.") from e # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='tuf-repo-cdn.sigstore.dev', port=443): Read timed out.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity
jku commentedon Jun 12, 2025
possibly in the streaming case only the connection establishment uses retries and the actual streaming does not?
ishaanxgupta commentedon Jan 4, 2026
Hi @jku can I work on this one?
jku commentedon Jan 5, 2026
feel free if you have ideas -- my assumption above was that
ngclient does use streaming. A modest number of retries would be nice but I would also like to avoid adding complicated code like manually restarting failed requests: so it may also be that there is no useful improvement to make here.