Add station-group stratification by lclanzi · Pull Request #259 · MeteoSwiss/evalml

lclanzi · 2026-09-07T15:14:56Z

This PR adds a new stratification axis, station group (all / holdout / holdin), alongside the existing region/season/init_hour stratifications, plus supporting infrastructure to make ground-truth retrieval more flexible. Note that this is a single fixed holdout/holdin split used for stratified evaluation, not k-fold cross-validation.

verification metrics can now be split by all / holdout / holdin stations, so skill can be evaluated separately on stations that were withheld from nudging (or other applications) vs. those that fed into it.
experiment.cross_validation in the top-level config (exclude_stations, or holdout_fraction + holdout_seed for a random split) is the single source of truth for which stations define the holdout group
verify() (src/verification/__init__.py) adds a station_group dimension to verification outputs, alongside the existing region/season/init_hour dimensions.
The dashboard (report_experiment_dashboard.py, template.html.jinja2, script.js) and scorecards (report_scorecard.py) are updated to expose and correctly aggregate this new dimension.
verif_hash() in common.smk now includes the station-group configuration, so changing it correctly invalidates cached verification outputs instead of silently reusing stale results.
Geographic station filtering for jretrieve-based truth: data_input.jretrieve.parse_selection() now supports filter_mode=switzerland (trim to stations within the real Swiss national border) and filter_mode=domain;domain_bbox=... (trim to an arbitrary bounding box), plus an optional use_limitation=<N> selector for data-usage-rights filtering. This lets a truth-root retrieve over a broad domain and then restrict to a precise region before verification

This PR depends on MeteoSwiss/anemoi-plugins-meteoswiss#40 and should be merged only afterwards.

…nd holdout

jonasbhend

Hi @lclanzi. Thanks for implementing this. I think the 'cross-validation' with holdin/out stations is a very nice feature. Given the additional way to select stations from jretrieve, I was wondering if we really want to support bbox selection and filtering down to Switzerland? What is the use case here that is not covered by selection by measurement network (e.g. 1,2) or bbox? If we don't have a strong argument, I would rather avoid this additional complexity.

Also, I am unable to check the implementation on linard, as mlflow is still being blocked. Will try to do a short test on Wednesday to confirm everything is working as expected.

lclanzi · 2026-09-08T06:59:31Z

Hi @lclanzi. Thanks for implementing this. I think the 'cross-validation' with holdin/out stations is a very nice feature. Given the additional way to select stations from jretrieve, I was wondering if we really want to support bbox selection and filtering down to Switzerland? What is the use case here that is not covered by selection by measurement network (e.g. 1,2) or bbox? If we don't have a strong argument, I would rather avoid this additional complexity.

Also, I am unable to check the implementation on linard, as mlflow is still being blocked. Will try to do a short test on Wednesday to confirm everything is working as expected.

Hi @jonasbhend, thanks a lot for the feedback!
I see your point, and it makes sense. However, I wasn’t able to retrieve SwissMetNet + partner stations (limited to Switzerland) directly through the jretrieve interface. Therefore, I currently retrieve all stations within the bounding box and filter them afterwards to keep only those in Switzerland. If you know how to achieve this directly with jretrieve, I will remove this additional complexity.

jonasbhend · 2026-09-08T09:35:36Z

I know that this doesn't work (easily), but my question is: why do you want to use only stations in Switzerland? I guess for the nudging, you wouldn't want to constrain, and for the evaluation we could argue similarly. If anything, I think for the evaluation we should implement it as a stratifiable region (in essence the hull of all forecasting regions), which would then be available for both station and analysis-based evaluations.

… for stratification

lclanzi · 2026-09-11T12:43:16Z

Thanks for the suggestions @jonasbhend!
I have added a new shapefile for Switzerland. This region is now available in the list and it is used for applying a region stratification over the country (instead of the filter in jretrieve)

dnerini

very cool feature! I left some comments

dnerini · 2026-10-05T12:12:45Z

is adding an experiment yaml really necessary for this new feature PR?

I removed the yaml. I will make a separate PR with varda-rapid related configs

looks like config's still there? or am I misunderstanding your comment?

dnerini · 2026-10-05T12:14:42Z

    return xr.Dataset(data_vars=data_vars, coords=coords)


+def _trim_stations_xr(


why is this necessary? can't be handled directly through the jretrieve query?

In principle, yes. However, I have not found a way to query SwissMetNet + partner stations within Switzerland using jretrieve. What I currently do is to query stations over a rectangular domain containing Switzerland with use_limitation=40, and further trim all stations located out of the Swiss borders. @clairemerker and @jonasbhend also looked into this, and If I remember correctly, you also had issues with this operation?

As mentioned previously, I don't see the point in doing the subsetting on the jretrieve part. For evalml it seems sufficient to use bbox selection in jretrieve together with region stratification (which now includes Switzerland only as a separate region, right)? So I still suggest to drop this to simplify and keep the loading of obs clean(ish).

As far as I know, bbox selection only works with a rectangular domain, meaning that stations out of Switzerland will be included in the ground truth. Or am I wrong?

No that is correct. bbox selection is what it is. But if you wanted stations only from Switzerland, there is probably ways to do that by selecting measurement networks. And in any case for verification you could just use bbox selection and then stratify by Switzerland as a region which should be equivalent to bbox selection and trimming to Switzerland on the jretrieve call. My question here is if there is a use case (for nudging I guess) where we would want to use only stations from Switzerland and not the surrounding areas (other than selecting SwissMetNet say).

Same, I also did not manage to make it work. And sometimes I can get more stations, but it is unclear to me which stations I am effectively retrieving. It would indeed be nice to have a better documentation or a more detailed explanation about the jretrieve API

I think stn_group_id doesn't provide enough options to retrieve the partner stations in Switzerland, at least I didn't manage to do so (either not implemented or not documented?). Maybe using meas_group_id could work, but the list is overwhelming to me, we will need help from MD to understand what to retrieve... (You can have a look at the options further down on that page: https://service.meteoswiss.ch/jretrieve/api/v1/surface/help)

Hi @clairemerker @lclanzi, I seem to be able to retrieve all automated stations by setting jretrieve:1,2,3,4 which is ['SMN', 'SMN precipitation', 'Partner', 'Partner precipitation']. Is this what you want?

Hm... I'll check again, it all got me so confused :D Thanks!

I feel the key question asked above by jonas is still valid though?

is there a use case where we would want to use only stations from Switzerland and not the surrounding areas?

my understanding is that we don't have such requirements for any of our use cases, do you agree? if this is true, the code above can be simplified

Thanks @jonasbhend, that works. However, I'm still unable to retrieve partner stations out of Switzerland. I can only do that by using the bbox selection in combination with use_limitation, which I believe gives little control to which stations are used (note that bbox and group are mutually exclusive)

dnerini · 2026-10-05T12:24:07Z

+        the config contains a ``forward_transform_filter:
+        nudge_toward_observation`` block, in which case its
+        ``exclude_stations``/``holdout_fraction``/``holdout_seed`` are
+        injected into that block — see ``_inject_nudging_station_holdout``.


I think I need to be convinced about this approach as it seems to me a bit too fragile...
Wouldn't be more robust to use the station_holdout_cfg to write a list of stations that is then used downstream, eg for nudging? I know the nudging filter expects a list of stations in its config, but maybe we could change that a path to a list of stations?

The holdout set is now a fixed list of station abbreviations in config/holdout_station/holdout_station.yaml. The evalml config points to this file with experiment.station_holdout_list. Verification uses it to split the stations that are in the truth dataset into holdout and holdin groups, for every model and baseline.

The nudging filter reads the same file through its holdout_station_file parameter and leaves those stations out when it corrects the initial condition. The file is read when the config is loaded.

I have to apologize, I realize I didn't explain myself properly above. I created a draft PR #273 and asked Claude in more detail to implement what I had in mind.

In #259, my review comment on `4658375` was about how the setting reached the nudging plugin. `inference_prepare.py` walked the inference YAML for a `nudge_toward_observation` block and wrote `exclude_stations` / `holdout_fraction` / `holdout_seed` into it. With `holdout_fraction`, verification (in `compute_holdout_stations`) and the plugin each drew their own random sample, so nothing guaranteed both used the same stations. The suggestion was: use the experiment config to **write a list of stations once**, then have the downstream steps (verification, nudging) consume that file. Commit `c83345d` read the comment differently. It moved the *source* out of the experiment config into a hand-written YAML file (`experiment.station_holdout_list: /abs/path.yaml`). The nudging config now points at that file separately with `holdout_station_file`. The workflow neither writes nor tracks the file. Remaining problems: - The same path has to be set in two places (experiment config and inference resource YAML), and nothing checks that they match. - The file isn't in the repo (`config/holdout_station/holdout_station.yaml` doesn't exist), so experiments aren't self-contained. - Snakemake doesn't know the file exists. Editing it reruns verification (through `VERIF_HASH`) but **not** inference, even though the nudged forecasts depend on it. ## Target design ``` experiment.station_holdout: [CHM, FRE, ...] (evalml config, single source) │ ▼ rule station_holdout ──► OUT_ROOT/data/station_holdout/{HOLDOUT_HASH}.csv (nat_abbr column) │ │ ▼ ▼ verification_metrics(_baseline) inference_prepare_* (only runs whose input: holdout=… inference config references holdout_stations.csv) --holdout_stations_file {input} copies → {workdir}/holdout_stations.csv │ ▼ nudge_toward_observation: holdout_station_file: holdout_stations.csv ``` The data passes through a declared rule output, so the DAG tracks it and every run gets a copy of exactly what it used. ### What changes - **Config:** `experiment.station_holdout` is an inline list of `nat_abbr`. Empty lists and duplicate stations are rejected. This replaces `station_holdout_list`. - **New rule `station_holdout`** (`data.smk`): writes `data/station_holdout/{hash}.csv` with a single `nat_abbr` column. Changing the list gives a new file, so everything that depends on it reruns. - **Verification:** `verification_metrics` and `verification_metrics_baseline` take the CSV as an input and pass it with `--holdout_stations_file`. - **Inference:** a prepare rule takes the CSV as an input only if its run's inference config mentions `holdout_stations.csv`. The prepare script copies it into the working directory under that name. Other models don't rerun when the holdout changes. - **Nudging config:** `holdout_station_file: holdout_stations.csv`, a relative path like the existing `obs_path`. - **Fail-fast:** the workflow stops at startup if an inference config references `holdout_stations.csv` but `experiment.station_holdout` is not set. ### Requires The nudging plugin's `holdout_station_file` must read a CSV with a `nat_abbr` column, relative to the working directory. I believe this is already implemented by MeteoSwiss/anemoi-plugins-meteoswiss@aeb2fcd. To be confirmed!

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Luca Lanzilao and others added 17 commits July 2, 2026 10:22

add config for nudging

bc5373a

update configs

6e13d16

update config and small edits

a81103c

update configs for nudging

6968349

chore: fix pre-commit

930ef89

add cross validation for nudging

9e43c40

fix bug in dashboard plotting

9f48857

small edits to dashboard

5b88610

add cross validation functionality: stations grouped in all, holdin a…

8c4bb45

…nd holdout

update configs

91b4c26

change selector layout in dashboard

b5c41df

edit configs

7ffe364

update cross validation methodology, add filtering to jretrieve output

0572149

Merge branch 'main' into feat/nudge-initial-condition-toward-observation

ce68e03

fix linting

3e8feb8

restore varda-single config

65e611f

fix linting

133e3cf

lclanzi requested review from dnerini and jonasbhend September 7, 2026 15:29

jonasbhend reviewed Sep 8, 2026

View reviewed changes

Comment thread src/data_input/__init__.py Outdated

lclanzi added 4 commits September 10, 2026 14:19

update configs

5955d8a

remove filtering from jretrieve and insert a new region (switzerland)…

e97998b

… for stratification

Merge branch 'main' into feat/nudge-initial-condition-toward-observation

21e480c

fix linting

24b5b02

dnerini reviewed Sep 14, 2026

View reviewed changes

Luca Lanzilao added 2 commits September 22, 2026 09:45

reply reviewer comments

460deea

Merge branch 'main' into feat/nudge-initial-condition-toward-observation

8ca4ed5

Luca Lanzilao added 4 commits September 22, 2026 09:59

fix linting

f662266

disentangle nudging logic from station stratification

88ee5c7

fix linting

4d62b4c

fix jretrieve path in config

ec031eb

dnerini reviewed Sep 23, 2026

View reviewed changes

Comment thread src/data_input/jretrieve.py Outdated

remove reference to previous version in comments

4658375

lclanzi requested review from dnerini and jonasbhend October 1, 2026 11:37

dnerini reviewed Oct 5, 2026

View reviewed changes

lclanzi added 4 commits October 6, 2026 07:50

remove varda-rapid configs

f1cbf29

apply reviewer comments

834c9c8

change holdout set definition framework

c83345d

revert change in max-missing-fraction

938387f

dnerini mentioned this pull request Oct 9, 2026

Pass holdout stations between rules as a CSV file #273

Merged

dnerini and others added 4 commits October 9, 2026 17:48

update tests

0801221

Merge branch 'main' into feat/nudge-initial-condition-toward-observation

c50d575

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Stop tracking personal Varda-rapid experiment configs

80b2a6c

		return xr.Dataset(data_vars=data_vars, coords=coords)


		def _trim_stations_xr(

Conversation

lclanzi commented Sep 7, 2026 • edited by dnerini Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonasbhend left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lclanzi commented Sep 8, 2026

Uh oh!

jonasbhend commented Sep 8, 2026

Uh oh!

lclanzi commented Sep 11, 2026

Uh oh!

dnerini left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lclanzi Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dnerini Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lclanzi Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lclanzi commented Sep 7, 2026 •

edited by dnerini

Loading

lclanzi Oct 6, 2026 •

edited

Loading

dnerini Oct 9, 2026 •

edited

Loading

lclanzi Oct 9, 2026 •

edited

Loading