[icoscp-core] NumPy > 2.0 incompatibility

[olivierbonte]

Hello,

When running the example code given here, there seems to be a problem with the code for Numpy 2.0 and newer.
To solve this issue I think you could

1. Restrict to numpy<2.0 in https://github.com/ICOS-Carbon-Portal/data/blob/master/src/main/python/icoscp_core/pyproject.toml#L20
2. Update to the NumPy 2.0 convention as suggested under "message" in the error below (seems recommended).

System information:

• OS: Windows10
• Python installed as conda env create -f environment.yml with the environment.yml defined below:

name: icos_error_fix_env
channels:
  - conda-forge
dependencies:
  - python=3.10
  - matplotlib
  - jupyterlab
  - pip
  - pip:
    - icoscp

Specifically I got following error:

	"name": "AttributeError",
	"message": "`newbyteorder` was removed from the ndarray class in NumPy 2.0. Use `arr.view(arr.dtype.newbyteorder(order))` instead.",
	"stack": "---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[1], line 26
     23 merged_co2 = pd.DataFrame(columns=['TIMESTAMP', 'co2'])
     25 # batch-fetch the interesting columns and iterate through the results
---> 26 for dobj, arrs in data.batch_get_columns_as_arrays(se_co2_from_100, ['TIMESTAMP', 'co2']):
     27     st_uri = dobj.station_uri
     28     # ICOS atmospheric station URIs end with underscore followed by a 3-letter station ID
     29     # this ID is convenient to use as a suffix to rename 'co2' with

File c:\\Users\\obonte\\AppData\\Local\\miniconda3\\envs\\icos_error_fix_env\\lib\\site-packages\\icoscp_core\\dataclient.py:228, in DataClient.batch_get_columns_as_arrays(self, dobjs, columns, keep_bad_data)
    225 \treport_cpb_file_read(self._conf, hashes=hashes, columns=columns)
    227 for dobj, codec in dobj_codecs:
--> 228 \tyield dobj, self._get_columns_as_arrays(codec, keep_bad_data)

File c:\\Users\\obonte\\AppData\\Local\\miniconda3\\envs\\icos_error_fix_env\\lib\\site-packages\\icoscp_core\\dataclient.py:241, in DataClient._get_columns_as_arrays(self, codec, keep_bad_data)
    239 url = self._conf.data_service_base_url + '/cpb'
    240 resp = http_request(url, \"Fetching binary from \" + url, \"POST\", headers, codec.json_payload)
--> 241 return codec.parse_cpb_response(resp.fp, keep_bad_data)

File c:\\Users\\obonte\\AppData\\Local\\miniconda3\\envs\\icos_error_fix_env\\lib\\site-packages\\icoscp_core\\cpb.py:191, in Codec.parse_cpb_response(self, resp, keep_bad_data)
    190 def parse_cpb_response(self, resp: io.BufferedReader, keep_bad_data: bool) -> ArraysDict:
--> 191 \treturn self._parse_from_buff(resp, None, keep_bad_data)

File c:\\Users\\obonte\\AppData\\Local\\miniconda3\\envs\\icos_error_fix_env\\lib\\site-packages\\icoscp_core\\cpb.py:225, in Codec._parse_from_buff(self, buff, col_offsets, keep_bad_data)
    221 \tarr = _type_post_process(arr, col.value_format_uri)
    223 \tif sys.byteorder == 'little':
    224 \t\t# cpb files are all big-endian, fixing for Pandas compatibility on little endian systems:
--> 225 \t\tarr = arr.byteswap().newbyteorder()
    226 \tres[col.label] = arr
    228 if not keep_bad_data and self._ci.good_flags:

AttributeError: `newbyteorder` was removed from the ndarray class in NumPy 2.0. Use `arr.view(arr.dtype.newbyteorder(order))` instead."

I tried to make a pull request myself, but I did not manage to set up the icoscp_core package in development mode (I tried using flit). Therefore, a bit more information on how to contribute (e.g. under https://github.com/ICOS-Carbon-Portal/data/tree/master/src/main/python/icoscp_core#getting-started) would be nice.

Thank you in advance,
Kind regards
Olivier Bonte

[mirzov]

Hi Olivier,

Thanks for the fantastic bug report! I will opt for solution 1 for now, as our higher-level lib depends on pandas, and that one depends on numpy 1.x (and probably will take a while to upgrade to 2.x).

[olivierbonte]

No worries, thanks for the quick fix!