Make fromager more resilient against network hickups

[tiran]

Fromager does not recover when there is a temporary network hickup. Example:

('Connection broken: IncompleteRead(16777216 bytes read, 86427880 more expected)', IncompleteRead(16777216 bytes read, 86427880 more expected))
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/response.py", line 748, in _error_catcher
    yield
  File "/usr/local/lib/python3.11/site-packages/urllib3/response.py", line 894, in _raw_read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
urllib3.exceptions.IncompleteRead: IncompleteRead(16777216 bytes read, 86427880 more expected)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 820, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
  File "/usr/local/lib/python3.11/site-packages/urllib3/response.py", line 1060, in stream
    data = self.read(amt=amt, decode_content=decode_content)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/response.py", line 949, in read
    data = self._raw_read(amt)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/response.py", line 872, in _raw_read
    with self._error_catcher():
  File "/usr/lib64/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/urllib3/response.py", line 772, in _error_catcher
    raise ProtocolError(arg, e) from e
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(16777216 bytes read, 86427880 more expected)', IncompleteRead(16777216 bytes read, 86427880 more expected))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/fromager/__main__.py", line 178, in invoke_main
    main(auto_envvar_prefix="FROMAGER")
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fromager/commands/download_sequence.py", line 52, in download_sequence
    sources.download_source(wkctx, req, [sdist_server_url])
  File "/usr/local/lib/python3.11/site-packages/fromager/sources.py", line 46, in download_source
    download_details = overrides.invoke(
                       ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fromager/overrides.py", line 59, in invoke
    return fn(**kwargs)
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fromager/sources.py", line 160, in default_download_source
    source_filename = _download_source_check(ctx.sdists_downloads, url, rename_to)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fromager/sources.py", line 172, in _download_source_check
    source_filename = download_url(destination_dir, url, rename_to)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fromager/sources.py", line 206, in download_url
    for chunk in r.iter_content(chunk_size=1024 * 1024):
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 822, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(16777216 bytes read, 86427880 more expected)', IncompleteRead(16777216 bytes read, 86427880 more expected))

[rd4398]

I researched a bit and found this : https://www.raelldottin.com/2023/09/enhancing-resilience-in-your-python.html
Does this approach make sense? Worth a shot?

[dhellmann]

@rd4398 using the retry feature makes sense. Could you create a module to make it easy to use that approach in the few places within fromager that we use requests? We probably also want to make it available for use in plugins for downloading things there.

[rd4398]

okay!