[NCI `fs38` catalog] what does `file_type` = `f` vs `l` mean and which is the correct one to use?

[Thomas-Moore-Creative]

on Intake-ESM on CMIP5/6 datasets the metadata file_type shows up but then just vanishes without explanation?

What does "l" vs "f" mean? And what to use? And why?

[Thomas-Moore-Creative]

response from NCI:

The ‘f’ and ‘l’ values of ‘file_type’ indicate whether the file is a regular file or a symbolic link. In some cases, a new revision may create a link to a file from a previous revision to save storage space.

[Thomas-Moore-Creative]

further response from NCI:

Soft links may exist in multiple versions, so we can’t rely on ‘l’ to identify the latest version, requiring several steps to determine it.
I agree that the catalog file should offer a more straightforward way to access the ‘latest’ dataset version. We’re working on adding ‘latest’ to the ‘versions’ column to make it easier to filter. I’ll let you know as soon as the updated catalog file is available for you to try.

[Thomas-Moore-Creative]

Documented some of this here:
https://forum.access-hive.org.au/t/cmip6-data-analysis-at-nci-file-type-issue/3842

[Thomas-Moore-Creative]

CLEX intake catalogs are still available and useful for CMIP5 and CMIP6 collections at NCI and suggest we make use of these until NCI updates their approach:

https://github.com/coecms/nci-intake-catalogue/blob/main/docs/intake_cmip6_demo.ipynb

clex_esgf_cat = intake.cat.nci['esgf']
clex_cmip6_cat = clex_esgf_cat['cmip6']
search_dict = {'experiment_id': ['piControl'],
      'source_id': 'ACCESS-ESM1-5',
      'variable_id': ['intpp']}
search = clex_cmip6_cat.search(**search_dict)

what is "clex_cmip6_cat"

print(type(clex_cmip6_cat))
`<class 'intake_esm.core.esm_datastore'>`

[Thomas-Moore-Creative]

Waiting for NCI to fix their fs38 catalog and using frozen CLEX catalog for now.

[Thomas-Moore-Creative]

With the end of CLEX and the death of hh5 we now need to deal with this.