Recipe testing for release 2.8.0 - Core release candidate rc1

[remi-kazeroni]

This issue documents the round of recipe testing performed using the Core release candidate v2.8.0rc1.

Release process

• Tool: ESMValTool v2.8 release #3055
• Core: ESMValCore v2.8 release ESMValCore#1924

System and settings

conda/mamba

(base) mamba --version
mamba 1.3.1
conda 23.1.0

Git branches and state

Tue 7 Mar 16:06:43 CET 2023

(base) :~/ESMValTool 
$ git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

(base) :~/ESMValCore 
$ git status
On branch v2.8.x
Your branch is up to date with 'origin/v2.8.x'.

nothing to commit, working tree clean

Installation and environment

$ cd ~/ESMValTool
$ mamba env create -n tool_280rc1 -f environment.yml
$ conda activate tool_280rc1
$ pip install --editable '.[develop]'
$ cd ~/ESMValCore
$ pip install --editable '.[develop]'

Config user file

Main options: all default except search_esgf: when_missing

output_dir: ./esmvaltool_output
max_parallel_tasks: 8
log_level: debug
exit_on_warning: false
output_file_type: png
remove_preproc_dir: true
compress_netcdf: false
save_intermediary_cubes: false
config_developer_file: null
profile_diagnostic: false

# Site-specific entries: DKRZ-Levante
search_esgf: when_missing
download_dir: /work/bd0854/DATA/ESMValTool2/download
auxiliary_data_dir: /work/bd0854/DATA/ESMValTool2/AUX
rootpath:
  CMIP6: /work/bd0854/DATA/ESMValTool2/CMIP6_DKRZ
  CMIP5: /work/bd0854/DATA/ESMValTool2/CMIP5_DKRZ
  CMIP3: /work/bd0854/DATA/ESMValTool2/CMIP3
  CORDEX: /work/ik1017/C3SCORDEX/data/c3s-cordex/output
  OBS: /work/bd0854/DATA/ESMValTool2/OBS
  OBS6: /work/bd0854/DATA/ESMValTool2/OBS
  obs4MIPs: /work/bd0854/DATA/ESMValTool2/OBS
  ana4mips: /work/bd0854/DATA/ESMValTool2/OBS
  native6: /work/bd0854/DATA/ESMValTool2/RAWOBS
  RAWOBS: /work/bd0854/DATA/ESMValTool2/RAWOBS
drs:
  CMIP6: DKRZ
  CMIP5: DKRZ
  CMIP3: DKRZ
  CORDEX: BADC
  obs4MIPs: default
  ana4mips: default
  OBS: default
  OBS6: default
  native6: default

ESMValTool version

$ esmvaltool version
ESMValCore: 2.8.0rc1
ESMValTool: 2.8.0.dev77+gd0c0c038e

Environment file

tool_280rc1.txt

Compute resources used

On DKRZ-Levante

• When possible (small to medium jobs, in practice 4 jobs per node):

#SBATCH --partition=interactive
#SBATCH --mem=64G

• Large jobs (nodes not shared)

#SBATCH --partition=compute

Note: no output comparison will be done for this round of testing. The main purpose is to identify issues with the Core and start fixing failing diagnostics. Results and overview webpage will be posted in a later post.

[remi-kazeroni]

Overview of the results

Numbers of successes and failures

The first round of recipe testing produced:

• 115 successful runs (i.e. where a index.html file was generated)
• 28 failed runs:
  • 7 for missing data/files
  • 13 diagnostic issues, including:
    • 8 failures related to matplotlib recent unpinning and upgrade in our environment Unpin matplotlib #3068
    • 4 failures related to perfmetrics/main.ncl
    • 1 failure related to numpy deprecation
  • 8 preprocessor issues:
    • several could be due to ESGF facet issues (recipes tested with search_esgf: when_missing)
    • recipe(s) requiring adaptation after Support wildcards in the recipe and improve support for ancillary variables and dataset versioning ESMValCore#1609
    • nothing critical at first sight!

For comparison, we released ESMValTool 2.7.0 with 4 non-working recipes (this could have been 8 if we used a stricter policy on missing data as done for this round of testing)

Overview webpage and path to data

• Webpage: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.8.0rc1/
• Path to the runs on Levante (with preproc dirs for failed runs): /work/bd0854/b309192/recipe_testing/recipe_testing_v2p8/v08rc1/scripts/esmvaltool_output

Note: I will now write three more posts to this issue for each category of failures (missing data, diagnostic issues, preprocessor issues) and provide more details about failures and lists of affected recipes. Please do not use this issue to comment on single recipe failures but rather open a separate issue, linking to this one. This issue is intended to provide a general overview or discuss recurring problems. More info and tagging the community soon 👍

[valeriupredoi]

@remi-kazeroni great work on this, bud! I see some manifestations of #3061 - so that might not be just a mocking test error; maybe @bouweandela has some time to look at, at least, anav failure 🍺

[remi-kazeroni]

Recipe failures due to missing data or files or data issues

Recipe	Test run	Problem	Assigned to	Related issue PR
recipe_esacci_lst	link	new ancestor taks issue due to malformed CMORized obs	-	#3104
recipe_flato13ipcc	link	known missing data issue	@katjaweigel	#2156
recipe_tebaldi21esd	link	new missing data issue: pr-tas, Amon, CMIP6, MPI-ESM1-2-HR, ScenarioMIP, ssp370, r*	@remi-kazeroni	#3093
recipe_autoassess_landsurface_soilmoisture	link	known missing climatology files (non-public)	@alistairsellar @valeriupredoi	marked as broken in #3103
recipe_schlund20jgr_gpp*	link to one	known retracted data issue	@schlunma	successfully tested with an extended data pool

• For recipe_collins13ipcc and recipe_tebaldi21esd, I would encourage recipe maintainers and friends to open dedicated issues to discuss the problem there and link here by editing the tables.

• For recipe_autoassess_landsurface_soilmoisture and recipe_schlund20jgr_gpp*, I would strongly recommend to discuss in the issue Application of the broken recipe policy for releases #3065 and not here.

[remi-kazeroni]

Recipe failures due to diagnostic errors

Recipe	Test run	Problem	Assigned to	Related issue PR
recipe_arctic_ocean	link	matplotlib issue	-	#3112
recipe_autoassess_stratosphere	link	matplotlib issue	-	#3088
recipe_bock20jgr_fig_8-10	link	matplotlib issue	@schlunma	fixed by #3087
recipe_carvalhais14nat	link	matplotlib issue	-	-
recipe_hype	link	numpy issue	@Peter9192	#3101
recipe_monitor	link	matplotlib issue	@sloosvel	see #3085
recipe_perfmetrics_CMIP5*	link to one	issue with perfmetrics/main.ncl	@LisaBock	#3083 #3098
recipe_smpi*	link to one	issue with perfmetrics/main.ncl	-	#3098
recipe_sea_surface_salinity	link	matplotlib issue	-@sloosvel	see #3100

• All these are new issues discovered in the release process of v2.8.0. I would encourage recipe maintainers and friends to open dedicated issues to discuss the problem there and link here by editing the tables.

• There are different types of "matplotlib issues". These are issues still existing after Replace removed matplotlib.pyplot.savefig option additional_artists #3075 was merged and used for the testing.

• If needed some problems could be moved to the preprocessor issues table (recipe_bock20jgr_fig_1-4 and recipe_esacci_lst)

[remi-kazeroni]

Recipe failures due to preprocessor errors

Recipe	Test run	Problem	Assigned to	Related issue PR
recipe_anav13jclim	link	issue with weighting_landsea_fraction for OBS data	@schlunma	run fine with #3064
recipe_autoassess_radiation_rms_cfMon_all	link	known CMOR check issue with CMOR checks	-	ESMValGroup/ESMValCore#1238 but won't fix for v2.8?
recipe_bock20jgr_fig_1-4	link	preproc issue reading OBS_HadCRUT4	-	ESMValGroup/ESMValCore#1962
recipe_check_obs	link	known derivation issue for ERA5	-	see ESMValGroup/ESMValCore#1388
recipe_climate_change_hotspot	link	new from_datasets.py issue	@sloosvel	#3086
recipe_collins13ipcc	link	new out of memory issue	-	#3106
recipe_meehl20sciadv	link	new _io.py concatenation issue	@schlunma	run fine with #3064
recipe_russell18jgr	link	new preprocessor/_supplementary_vars.py issue	@schlunma	run fine with #3064
recipe_schlund20esd	link	new _io.py concatenation issue	@schlunma	run fine with #3064
recipe_wenzel16jclim	link	new preprocessor/_supplementary_vars.py issue	@schlunma	#3084; run fine with #3064

• I have the impression that some (all?) the 4-5 recipes could be run successfully when using search_esgf: never, i.e. without ESGF data. Said differently, some of the listed preprocessor issues could be ESGF facet issues...

• The last 5 failures are new issues discovered in the release process of v2.8. I would encourage recipe maintainers and friends to open dedicated issues to discuss the problem there and link here by editing the tables.

[remi-kazeroni]

Hi @ESMValGroup/esmvaltool-developmentteam and @ESMValGroup/esmvaltool-recipe-maintainers, the results from the first round of recipe testing for the release of ESMValTool and ESMValCore v2.8 are now available. See summary and link to the overview webpage in this #3076 (comment) above.

It would be great if you could take a look at the results of your favourite recipes and help fixing the failing ones. In such cases, please assign yourselves in the tables above and if possible open separate issues to discuss the problems. Feel free to comment here if you have noticed other problems with the recipe output.

There are 3 main categories of failures:

• missing data issues
• diagnostic issues
• preprocessor issues

Any help will be greatly appreciated 👍
Please tag me in the PRs for recipe fixing, I can take a look and merge them into the main and release branches. Thanks in advance for your support!

[valeriupredoi]

@remi-kazeroni one thing you should tell the users that will be running their recipes is to deactivate the esmvalcore 2.7.* pin in environment.yml and make it either 2.8.* or remove it completely, and to activate the rc channel 👍

[valeriupredoi]

@remi-kazeroni one thing you should tell the users that will be running their recipes is to deactivate the esmvalcore 2.7.* pin in environment.yml and make it either 2.8.* or remove it completely, and to activate the rc channel +1

hang on, we have an issue with the Tool env picking up Core 2.8.0rc1 off the rc channel/label - am looking into this atm 👍