Add a directory and filename template with facets stored in directory for OBS and OBS6?

[bouweandela]

At the moment, all facets are contained in the filename instead of in the directory for OBS and OBS6 data. This prevents the facets from being read from the directory name. Therefore wildcards in the recipe do not work as well as expected for these projects, e.g. automatically adding supplementary variables does not work. See the section Backward incompatible changes in #1609 (comment).

We could add an extra directory and filename template to config-developer.yml that contains the facets in the directory instead of in the file. For example, we could add a default2 (or a better name if someone knows it), that is the same as for native6:

OBS:
  cmor_strict: false
  input_dir:
    default: 'Tier{tier}/{dataset}'
    default2: 'Tier{tier}/{dataset}/{version}/{frequency}/{short_name}'
    BSC: '{type}/{institute.lower}/{dataset.lower}/{freq_folder}/{short_name}{freq_base}'
    RCAST: '{dataset}'
  input_file:
    default: '{project}_{dataset}_{type}_{version}_{mip}_{short_name}[_.]*nc'
    default2: '*.nc'
    BSC: '{short_name}_*.nc'
    RCAST: '{short_name}_{mip}_{type}_{dataset}_*.nc'
  output_file: '{project}_{dataset}_{type}_{version}_{mip}_{short_name}'
  cmor_type: 'CMIP5'

OBS6:
  cmor_strict: false
  input_dir:
    default: 'Tier{tier}/{dataset}'
    default2: 'Tier{tier}/{dataset}/{version}/{frequency}/{short_name}'
    BSC: '{type}/{institute.lower}/{dataset.lower}/{freq_folder}/{short_name}{freq_base}'
  input_file:
    default: '{project}_{dataset}_{type}_{version}_{mip}_{short_name}[_.]*nc'
    default2: '*.nc'
    BSC: '{short_name}_*.nc'
  output_file: '{project}_{dataset}_{type}_{version}_{mip}_{short_name}'
  cmor_type: 'CMIP6'

native6:
  cmor_strict: false
  input_dir:
    default: 'Tier{tier}/{dataset}/{version}/{frequency}/{short_name}'
  input_file:
    default: '*.nc'
  output_file: '{project}_{dataset}_{type}_{version}_{mip}_{short_name}'
  cmor_type: 'CMIP6'
  cmor_default_table_prefix: 'CMIP6_'

That would allow users to set

drs:
  OBS: default2
  OBS6: default2

in their config-user.yml file and reorganize their OBS data accordingly and then automatic addition of supplementary files would work for the OBS and OBS6 projects without any changes to the recipe.

This would make it easier to upgrade if we add it to v2.8.0. If anyone likes this idea, please comment @ESMValGroup/esmvaltool-coreteam. If not, we can close the issue.

[valeriupredoi]

I believe @remi-kazeroni is probably in the best position to comment relevantly on this, and power users like @schlunma or @sloosvel - I like it, but am wondering what's the level of standardization when it comes to OBS data dirs' structures - we have them almost ESGF-like on JASMIN and DKRZ but I know in other locations they are just strewn around like tools in a workshop, Then again, this would be nice to have to get some order and add to the case of adding OBS'es to the ESGF. Then again 2, we're kinda pressed on time for 2.8 - so I'll defer to the RM 😁

[remi-kazeroni]

The idea is interesting but I think it will introduce lots of complications for our users who would have to restructure their entire OBS data pool to get affected recipes by #1609 running. I have no idea how many groups would have to do that but I could imagine this would be quite a few... I think this is too big of a change and asking users to restructure OBS data pool should be done with great care and only if no other solution is possible. This is one of the trickiest steps when setting up a shared ESMValTool eco-system on clusters...

Groups who would not go through that process would not be able to run recipes affected by #1609. I would really prefer that we try to fix affected recipes and enabling the Tool to read facets from filenames after curating problematic dataset names and versions. These changes would affect less users because not all datasets are impacted.

Also, the proposed changes lead to backward incompatible solutions and should be discussed for 2 weeks at least. I'm afraid we don't have this time before the release. I would suggest to first discuss this in a larger circle (workshop?) and also explore the possibility of reading facets from filenames.

[bouweandela]

I'll close the issue then and we can think about it later if it's still relevant.