`--search_data=quick` does not work as intended

[schlunma]

Describe the bug
--search_data=quick does not work as intended. The doc says:

When set to quick, search will stop as soon as a result is found.

This is not the case. For example, a recipe which requires fx variables fails if multiple versions of that fx file are available (e.g., from ESGF and locally).

However, setting --search_data=complete fixes this.

I guess the problem is here:

ESMValCore/esmvalcore/dataset.py

Lines 771 to 779 in 82b282f

	# 'quick' mode: if files are available from a higher
	# priority source, do not search lower priority sources.
	if self.session["search_data"] == "complete":
	try:
	check.data_availability(self, log=False)
	except InputFilesNotFound:
	pass # continue search for data
	else:
	return # use what has been found so far

Replacing complete with quick should do the trick.

[bouweandela]

Replacing complete with quick here seems to make sense, but the bug you describe should not be present regardless of which setting you're using for search_data. Could that be another issue?

[schlunma]

The question in this case is how we handle duplicate data.

For data with a time coordinate, we have some logic that removes duplicate time periods.

For data without a time coordinate (like fx files), we do not have that logic, and an error is raised because the tool attempts concatenation but the time coordinate is missing. I agree that the error message is not optimal, but I think we should raise an error here because we cannot tell which file is "better".

Replacing complete with quick here restores the original behavior, which basically ignores the downloaded data if the file is available locally.

[bouweandela]

We handle duplicate data with this code:

ESMValCore/esmvalcore/dataset.py

Lines 758 to 768 in 82b282f

	files: dict[str, DataElement] = {}
	for data_source in sorted(
	_get_data_sources(self.session, self.facets["project"]), # type: ignore[arg-type]
	key=lambda ds: ds.priority,
	):
	result = data_source.find_data(**self.facets)
	for file in result:
	if file.name not in files:
	files[file.name] = file
	if version(files[file.name]) < version(file):
	files[file.name] = file

so basically we go over the data sources by priority and keep only the first DataElement for each DataElement.name attribute, except when a DataElement with a newer version facet is found, then we use that instead. See also the note at the end of this section: https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#hpc-system

[schlunma]

But the note says that "Deduplicating data found via esmvalcore.io.intake_esgf data sources and the esmvalcore.io.local data sources has not yet been implemented"?

I think the problem here is that file.name is different for local and ESGF files:

Local file:

print(file)  # /work/ik1017/CMIP6/data/CMIP6/CMIP/BCC/BCC-ESM1/1pctCO2/r1i1p1f1/fx/areacella/gn/v20190613/areacella_fx_BCC-ESM1_1pctCO2_r1i1p1f1_gn.nc
print(file.name)  # areacella_fx_BCC-ESM1_1pctCO2_r1i1p1f1_gn.nc

ESGF file:

print(file)  # IntakeESGFDataset(name='CMIP6.CMIP.BCC.BCC-ESM1.1pctCO2.r1i1p1f1.fx.areacella.gn')
print(file.name)  # CMIP6.CMIP.BCC.BCC-ESM1.1pctCO2.r1i1p1f1.fx.areacella.gn

[schlunma]

Note that I had to set priority=2 in my configuration of the intake data source to restore the old behavior with search_data=quick in #2937.

[bouweandela]

Ah, I hadn't realized you were mixing these two data sources.

Indeed, support for mixing these has indeed not yet been implemented: esmvalcore.io.intake_esgf.IntakeESGFDataSource uses one name for the entire 'dataset' because there are plans to provide virtual zarr files through ESGF and those have been aggregated this way. To make this work, we would need to introduce a new intermediate object that represents a list of files found by esmvalcore.io.local.LocalDataSource (and esmvalcore.io.esgf.ESGFDataSource) that corresponds to a 'dataset'.

[bouweandela]

See https://api.stac.ceda.ac.uk/search and click through to features > 0 > assets > reference_file for an example of a virtual zarr file.