Config validation fails to discover Anemoi library path

[phinate]

What happened?

To discover the defaults for configs supplied by the package, Hydra needs

path=pkg://anemoi.training/config

in searchpath. This isn't picked up when running

anemoi training config validate --config-name=...

An example config is here:

defaults:
  - data: zarr
  - dataloader: native_grid
  - diagnostics: evaluation
  - datamodule: single
  - hardware: example
  - graph: multi_scale
  - model: gnn
  - training: default
  - _self_

config_validation: True

data:
  resolution: o96

hardware:
  num_gpus_per_node: 1
  accelerator: auto
  paths:
    data: https://object-store.os-api.cci1.ecmwf.int/ml-tests/test-data/samples
    output: outputs/ # anything writeable
    graph: outputs/graphs # downloads graph here if absent
  files:
    dataset: anemoi-integration-tests/training/datasets/aifs-ea-an-oper-0001-mars-o96-2017-2017-6h-v8-testing.zarr
    graph: first_graph_o96.pt

dataloader:
  training:
    end: 2017-01-08 12:00:00
  validation:
    start: 2017-01-08 18:00:00

training:
  max_epochs: 1
  lr:
    rate: 1e-3

diagnostics:
  log:
    wandb: { enabled: false, entity: "" }
    mlflow: { enabled: false, tracking_uri: "" }

Running anemoi-training train ... gives

[2025-09-30 14:19:55,592][anemoi.training.train.train][INFO] - Config validated.

as part of the logs. However, running anemoi-training config validate ... gives

hydra.errors.MissingConfigException: In 'simple-config.yaml': Could not find 'training/default'

We can further see the search path does not contain path=pkg://anemoi.training/config:

Config search path:
        provider=anemoi-cwd-searchpath-plugin, path=file:///Users/nsimpson/Code/temp-an
        provider=hydra, path=pkg://hydra.conf
        provider=main, path=file:///Users/nsimpson/Code/temp-an/.venv/lib/python3.12/site-packages/anemoi/training/commands
        provider=schema, path=structured://

Version

feat/azure-mlflow

Platform (OS and architecture)

MacOS 14.8

[anaprietonem]

@phinate can you share the output of the search path logs from running the anemoi-training train, or is it just as the one you posted but including the missing pkg://anemoi.training/config?

This could be related to how we are setting up hydra in the config validation command vs how it's done for the training command. @JesperDramsch do you remember how's the AnemoiSearchPathPlugin working to add path=pkg://anemoi.training/config to search_path? (does it depend on using hydra.main vs instantiate)

cc: @gmertes

[JesperDramsch]

Hi @phinate,

Apologies for the delay and thank you for providing such detailed analysis!

I have looked at the config validate command, which was using initialize(version_base=None, config_path="") which prevented Hydra from discovering the package configs. Normally, those were included automatically but weren't picked up here.

I looked at two fixes, one that would modify that path in the initialise and one that updates the AnemoiSearchPathPlugin to explicitly include the package over relying on any automatisms by hydra.

I've implemented a fix using the AnemoiSearchPathPlugin approach, which adds pkg://anemoi.training/config to the search path globally.

The fix has been tested with:

• Unit tests for the validate_config functionality
• Manual verification that configs referencing training/default now work correctly
• Confirmation that the search path includes the package config path

Happy to open a PR with the fix.

[JesperDramsch]

@phinate I opened a PR that fixes the issue for me. Would you mind checking if that works for you as well?

[phinate]

I can confirm that #656 puts the anemoi package defaults to the end of the search path. With the config at the top of the issue:

$ anemoi-training config validate --config-name=config.yaml
2025-11-19 19:33:13 INFO Validating configs.
2025-11-19 19:33:13 WARNING Note that this command is not taking into account if your config has set                     the config_validation flag to false.So this command will validate the config regardless of the flag.
2025-11-19 19:33:13 INFO Prepending current user directory (/Users/nsimpson/Code/anemoi-core/config) to the search path.
2025-11-19 19:33:13 INFO Search path is now: [
  provider=anemoi-cwd-searchpath-plugin, 
  path=/Users/nsimpson/Code/anemoi-core/config, 
  provider=hydra, path=pkg://hydra.conf, 
  provider=main, path=/Users/nsimpson/Code/anemoi-core/training/src/anemoi/training/commands, 
  provider=anemoi-package-searchpath-plugin, path=pkg://anemoi.training/config
]
2025-11-19 19:33:13 INFO Config files validated.

Thanks for this @JesperDramsch!