error on WV SLC product download (incomplet read)

[agrouaze]

this issue crash (randomly?) when downloading product for which donwload is quite long (IW or WV SLC for instance).

product in error:
8c695363-f650-434c-b465-8364e59f7481,S1A_WV_SLC__1SSV_20231201T064122_20231201T071521_051457_0635DA_ED84.SAFE

A traceback

01/12/2023 12:03:33 INFO download.py(491) start download : S1A_WV_SLC__1SSV_20231201T064122_20231201T071521_051457_0635DA_ED84.SAFE
CDSE download defaultdict(<class 'int'>, {'total_product_to_download': 9648, 'product_absent_from_local_disks': 329, 'archived_product': 9319}):   0%|                                  | 0/20 [24:41<?, ?it/s]
Traceback (most recent call last):
  File "/opt/conda-envs/dev/lib/python3.10/site-packages/urllib3/response.py", line 710, in _error_catcher
    yield
  File "/opt/conda-envs/dev/lib/python3.10/site-packages/urllib3/response.py", line 835, in _raw_read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
urllib3.exceptions.IncompleteRead: IncompleteRead(4575162672 bytes read, 8620036116 more expected)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda-envs/dev/lib/python3.10/site-packages/requests/models.py", line 816, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
  File "/opt/conda-envs/dev/lib/python3.10/site-packages/urllib3/response.py", line 940, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/opt/conda-envs/dev/lib/python3.10/site-packages/urllib3/response.py", line 911, in read
    data = self._raw_read(amt)
  File "/opt/conda-envs/dev/lib/python3.10/site-packages/urllib3/response.py", line 813, in _raw_read
    with self._error_catcher():
  File "/opt/conda-envs/dev/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/conda-envs/dev/lib/python3.10/site-packages/urllib3/response.py", line 727, in _error_catcher
    raise ProtocolError(f"Connection broken: {e!r}", e) from e
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(4575162672 bytes read, 8620036116 more expected)', IncompleteRead(4575162672 bytes read, 8620036116 more expected))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home1/datahome/agrouaze/sources/git/cdsodatacli/tests_metiers/test_download_WV.py", line 66, in <module>
    dfout = download_list_product_sequential(
  File "/home1/datahome/agrouaze/sources/git/cdsodatacli/cdsodatacli/download.py", line 529, in download_list_product_sequential
    ) = CDS_Odata_download_one_product_v2(
  File "/home1/datahome/agrouaze/sources/git/cdsodatacli/cdsodatacli/download.py", line 110, in CDS_Odata_download_one_product_v2
    for chunk in response.iter_content(chunk_size=8192):
  File "/opt/conda-envs/dev/lib/python3.10/site-packages/requests/models.py", line 818, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(4575162672 bytes read, 8620036116 more expected)', IncompleteRead(4575162672 bytes read, 8620036116 more expected))

it is not clear whether it is an issue coming from cdsodatacli or from the OData API, a ticket is open in support system CDSE:

[agrouaze]

test using curl:

curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalogue.dataspace.copernicus.eu/odata/v1/Products(8c695363-f650-434c-b465-8364e59f7481)/$value' --location-trusted --output /tmp/product.zip

no problem of download using curl:

 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   162  100   162    0     0    794      0 --:--:-- --:--:-- --:--:--   794
100 12.2G  100 12.2G    0     0  7070k      0  0:30:22  0:30:22 --:--:-- 5485k
/tmp/product.zip: Zip archive data, at least v2.0 to extract, compression method=store

So the issue is not coming from OData server but from the python request implementation.