Skip to content

Commit

Permalink
feat(nc2np): improvement to handle large files
Browse files Browse the repository at this point in the history
Re-introducing 'chunks="auto"' to be able to load files in the GB range,
where we now drop unwanted variables before opening the file.
  • Loading branch information
engeir committed Sep 12, 2023
1 parent 9146205 commit 1c3f9c3
Showing 1 changed file with 2 additions and 3 deletions.
5 changes: 2 additions & 3 deletions src/cesm_helper_scripts/nc_to_np.py
Original file line number Diff line number Diff line change
Expand Up @@ -148,15 +148,14 @@ def nc_to_np(temps: xr.Dataset):

def main():
"""Run the main function for the script."""
array_ds = xr.open_mfdataset(inputs)
array_ds = xr.open_mfdataset(inputs, chunks="auto", drop_variables="time_bnds")
attr_list = list(array_ds.data_vars)
if "time_bnds" in attr_list:
attr_list.remove("time_bnds")
if len(attr_list) != 1:
raise ValueError(
"The input file must contain only one variable. " + f"Found {attr_list}"
)
array = getattr(array_ds, attr_list[0]).assign_attrs(array_ds.attrs)
array_ds.close()
nc_to_np(array)


Expand Down

0 comments on commit 1c3f9c3

Please sign in to comment.