feat: add config validation and enable config key deletion

[coroa]

Changes proposed in this Pull Request

We are encountering during extension of pypsa-eur in soft-forks that we cannot account for all possible config options. Therefore i am proposing a central place in maybe scripts/config.py > validate_config where one would check for consistency of configuration settings.

This validate_config function is called from scenario_config for config providers and for the initial config (at the top of Snakefile), it is expected to raise an exception when there is a conflict.

At the moment the function is a stub, a good schema with a detailed report of all the failed validations should be added before this is merged. But i would like to get a quick idea of whether you think a mechanism like that is valid? This might also be a good place for a jsonschema or pydantic model evaluation?

The second problem this PR is addressing is that the config is recursively pruned from null values so that one can remove keys from dictionaries in higher priority configs.

@tgilon @daniel-rdt

Checklist

• I tested my contribution locally and it works as intended.
• Code and workflow changes are sufficiently documented.
• Changed dependencies are added to envs/environment.yaml.
• Changes in configuration options are added in config/config.default.yaml.
• Changes in configuration options are documented in doc/configtables/*.csv.
• Sources of newly added data are documented in doc/data_sources.rst.
• A release note doc/release_notes.rst is added.

[coroa]

maybe we would want to validate here as well? in principle the DAG creation should have validated already, so i decided against it.

[lkstrp]

In theory, extending the pruning could make the configuration invalid, couldn't it? Which means it would be best to validate the configuration whenever it is changed

[coroa]

During DAG generation the same pruning is also applied and then validated, so i think it would still be fine.

[lkstrp]

Ref #1547

[coroa]

Ref #1547

Very true, exactly the same concept, should have reviewed again. Only thing that is new here is null-value pruning and application to scenario_config.

What is the state there?

[lkstrp]

The state is pretty much still as described in the PR. JSON Schema is quite verbose, difficult to set up, and unable to handle complicated validations. Since pydantic now supports full export to JSON Schema, I would switch to a minimal pydantic implementation, which we could then extend. The issue with the minimal implementation is that we cannot replace all the doc tables at once.

I'm not sure if pruning should be done with Pydantic as well. But could be moved also later. Otherwise, if you're only defining a placeholder here to have something for soft forks, go ahead

[coroa]

implementation, which we could then extend. The issue with the minimal implementation is that we cannot replace all the doc tables at once.

I'm not sure if pruning should be done with Pydantic as well. But could be moved also later. Otherwise, if you're only defining a placeholder here to have something for soft forks, go ahead

Thanks, indeed.

This here is mainly thought of as a placeholder where soft-forks can add custom validation functions. Good to know about pydantic as new plan and i prefer the lib/validation.py placement you introduced over in the other PR.

Will adapt here.

[fneum]

To be closed by #1912.