Check for zeros in the data

[peterdudfield]

Detailed Description

It would be great to have a check in place that checks for zeros. A large amount of these is normally an error

Context

• good to catch data problems early and fail hard

Possible Implementation

• add a a check, if number os zeros >20% of the data, error
• @devsjc can you suggest a place in the code for this check?

[devsjc]

Yes, I would add it to the filter here:

nwp-consumer/src/nwp_consumer/internal/service/consumer.py

Lines 410 to 424 in 372359c

	def _dataQualityFilter(ds: xr.Dataset) -> bool:
	"""Filter out data that is not of sufficient quality."""
	if ds == xr.Dataset():
	return False
	# Carry out a basic data quality check
	for data_var in ds.data_vars:
	if ds[f"{data_var}"].isnull().any():
	log.warn(
	event=f"Dataset has NaNs in variable {data_var}",
	initTime=str(ds.coords["init_time"].values[0])[:16],
	variable=data_var,
	)
	return True