Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update documentations of Configuration default environments #3321

Merged
merged 21 commits into from
Dec 7, 2023
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 62 additions & 1 deletion docs/source/configuration/advanced_configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

By default, Kedro is set up to use the [OmegaConfigLoader](/kedro.config.OmegaConfigLoader) class.

## Advanced configuration for Kedro projects
This page also contains a set of guidance for advanced configuration requirements of standard Kedro projects:

* [How to use a custom config loader](#how-to-use-a-custom-configuration-loader)
Expand All @@ -17,7 +18,7 @@
* [How to change the merge strategy used by `OmegaConfigLoader`](#how-to-change-the-merge-strategy-used-by-omegaconfigloader)


## How to use a custom configuration loader
### How to use a custom configuration loader
You can implement a custom configuration loader by extending the [`AbstractConfigLoader`](/kedro.config.AbstractConfigLoader) class:

```python
Expand Down Expand Up @@ -74,6 +75,10 @@
### How to bypass the configuration loading rules
You can bypass the configuration patterns and set configuration directly on the instance of a config loader class. You can bypass the default configuration (catalog, parameters and credentials) as well as additional configuration.

For example, you can [use hooks to load external credentials](../hooks/common_use_cases.md#use-hooks-to-load-external-credentials).

Alternatively, if you are using config loader as a standalone component, you can override configuration as follows:

Check warning on line 81 in docs/source/configuration/advanced_configuration.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/source/configuration/advanced_configuration.md#L81

[Kedro.weaselwords] 'Alternatively' is a weasel word!
Raw output
{"message": "[Kedro.weaselwords] 'Alternatively' is a weasel word!", "location": {"path": "docs/source/configuration/advanced_configuration.md", "range": {"start": {"line": 81, "column": 1}}}, "severity": "WARNING"}

Check warning on line 81 in docs/source/configuration/advanced_configuration.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/source/configuration/advanced_configuration.md#L81

[Kedro.toowordy] 'Alternatively' is too wordy
Raw output
{"message": "[Kedro.toowordy] 'Alternatively' is too wordy", "location": {"path": "docs/source/configuration/advanced_configuration.md", "range": {"start": {"line": 81, "column": 1}}}, "severity": "WARNING"}
```{code-block} python
:lineno-start: 10
:emphasize-lines: 8
Expand Down Expand Up @@ -285,3 +290,59 @@

If no merge strategy is defined, the default destructive strategy will be applied. Note that this merge strategy setting only applies to configuration files in **different** environments.
When files are part of the same environment, they are always merged in a soft way. An error is thrown when files in the same environment contain the same top-level keys.


## Advanced configuration without a full Kedro project
In some cases, you may only want to use the `OmegaConfigLoader` without a Kedro project. By default, a Kedro project has a `base` and `local` environment.
However, when you use the `OmegaConfigLoader` directly, it assumes *no* environment. You may find it useful to [add Kedro to your existing notebooks](../notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md).

### Read configuration
The config loader can work without a Kedro project structure.
```bash
tree .
.
└── parameters.yml
```

```yaml
# parameters.yml
learning_rate: 0.01
train_test_ratio: 0.7
```

```python
from kedro.config import OmegaConfigLoader
config_loader = OmegaConfigLoader(conf_source=".")

Check warning on line 316 in docs/source/configuration/advanced_configuration.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/source/configuration/advanced_configuration.md#L316

[Kedro.words] Use '**Note**:' instead of 'Note that'.
Raw output
{"message": "[Kedro.words] Use '**Note**:' instead of 'Note that'.", "location": {"path": "docs/source/configuration/advanced_configuration.md", "range": {"start": {"line": 316, "column": 84}}}, "severity": "WARNING"}

Check warning on line 316 in docs/source/configuration/advanced_configuration.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/source/configuration/advanced_configuration.md#L316

[Kedro.weaselwords] 'only' is a weasel word!
Raw output
{"message": "[Kedro.weaselwords] 'only' is a weasel word!", "location": {"path": "docs/source/configuration/advanced_configuration.md", "range": {"start": {"line": 316, "column": 122}}}, "severity": "WARNING"}
# Optionally, you can also use environments
# config_loader = OmegaConfigLoader(conf_source=".", base_env="base", default_run_env="local")

>>> config_loader["parameters"]
{'learning_rate': 0.01, 'train_test_ratio': 0.7}

Check warning on line 321 in docs/source/configuration/advanced_configuration.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/source/configuration/advanced_configuration.md#L321

[Kedro.weaselwords] 'only' is a weasel word!
Raw output
{"message": "[Kedro.weaselwords] 'only' is a weasel word!", "location": {"path": "docs/source/configuration/advanced_configuration.md", "range": {"start": {"line": 321, "column": 25}}}, "severity": "WARNING"}
```

Check warning on line 322 in docs/source/configuration/advanced_configuration.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/source/configuration/advanced_configuration.md#L322

[Kedro.toowordy] 'However' is too wordy
Raw output
{"message": "[Kedro.toowordy] 'However' is too wordy", "location": {"path": "docs/source/configuration/advanced_configuration.md", "range": {"start": {"line": 322, "column": 1}}}, "severity": "WARNING"}

For the full list of features, please refer to [configuration_basics](./configuration_basics.md) and [advanced_configuration](./advanced_configuration.md)

### How to use Custom Resolvers with `OmegaConfigLoader`
You can register custom resolvers to use non-primitive types for parmaeters.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You can register custom resolvers to use non-primitive types for parmaeters.
You can register custom resolvers to use non-primitive types for parameters.


```yaml
# parameters.yml
polar_float64: "${polars: Float64}"
astrojuanlu marked this conversation as resolved.
Show resolved Hide resolved
today: "${today:}"
```

```python
import polars as pl
from datetime import date

from kedro.config import OmegaConfigLoader

custom_resolvers = {"polars": lambda x: getattr(pl, x),
"today": lambda: date.today()}

# Register custom resolvers
config_loader = OmegaConfigLoader(conf_source=".", custom_resolvers=custom_resolvers)
>>> print(config_loader["parameters"])
{'polar_float64': Float64, 'today': datetime.date(2023, 11, 23)}
astrojuanlu marked this conversation as resolved.
Show resolved Hide resolved
```
4 changes: 2 additions & 2 deletions docs/source/configuration/configuration_basics.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ The configuration source folder is [`conf`](../get_started/kedro_concepts.md#con
## Configuration environments
A configuration environment is a way of organising your configuration settings for different stages of your data pipeline. For example, you might have different settings for development, testing, and production environments.

By default, Kedro has a `base` and a `local` environment.
By default, Kedro projects have a `base` and a `local` environment.

### Base
In Kedro, the base configuration environment refers to the default configuration settings that are used as the foundation for all other configuration environments.
Expand Down Expand Up @@ -60,7 +60,7 @@ Configuration files will be matched according to file name and type rules. Suppo
* *Either* of the following is true:
* filename starts with `catalog`
* file is located in a subfolder whose name is prefixed with `catalog`
* *And* file extension is one of the following: yaml`, `yml`, or `json`
* *And* file extension is one of the following: `yaml`, `yml`, or `json`

### Configuration patterns
Under the hood, the Kedro configuration loader loads files based on regex patterns that specify the naming convention for configuration files. These patterns are specified by `config_patterns` in the configuration loader classes.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -436,7 +436,7 @@
"source": [
"from kedro.config import OmegaConfigLoader\n",
"\n",
"conf_loader = OmegaConfigLoader(\".\", base_env=\"\", default_run_env=\"\")"
"conf_loader = OmegaConfigLoader(conf_source=\".\")"
]
},
{
Expand Down Expand Up @@ -548,7 +548,7 @@
"from kedro.config import OmegaConfigLoader\n",
"from kedro.io import DataCatalog\n",
"\n",
"conf_loader = OmegaConfigLoader(\".\", base_env=\"\", default_run_env=\"\")\n",
"conf_loader = OmegaConfigLoader(conf_source=\".\")\n",
"conf_catalog = conf_loader[\"catalog\"]\n",
"\n",
"# Create the DataCatalog instance from the configuration\n",
Expand Down Expand Up @@ -584,7 +584,7 @@
"from kedro.config import OmegaConfigLoader\n",
"from kedro.io import DataCatalog\n",
"\n",
"conf_loader = OmegaConfigLoader(\".\", base_env=\"\", default_run_env=\"\")\n",
"conf_loader = OmegaConfigLoader(conf_source=\".\")\n",
"conf_catalog = conf_loader[\"catalog\"]\n",
"conf_params = conf_loader[\"parameters\"]\n",
"\n",
Expand Down Expand Up @@ -804,7 +804,7 @@
"from kedro.config import OmegaConfigLoader\n",
"from kedro.io import DataCatalog\n",
"\n",
"conf_loader = OmegaConfigLoader(\".\", base_env=\"\", default_run_env=\"\")\n",
"conf_loader = OmegaConfigLoader(conf_source=\".\")\n",
"conf_catalog = conf_loader[\"catalog\"]\n",
"conf_params = conf_loader[\"parameters\"]\n",
"\n",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -284,7 +284,7 @@ To use Kedro's `OmegaConfigLoader` to load `parameters.yml` the code is as follo
```python
from kedro.config import OmegaConfigLoader

conf_loader = OmegaConfigLoader(".", base_env="", default_run_env="")
conf_loader = OmegaConfigLoader(conf_source=".")
```

```python
Expand Down Expand Up @@ -348,7 +348,7 @@ To load `catalog.yml` the code is as follows:
from kedro.config import OmegaConfigLoader
from kedro.io import DataCatalog

conf_loader = OmegaConfigLoader(".", base_env="", default_run_env="")
conf_loader = OmegaConfigLoader(conf_source=".")
conf_catalog = conf_loader["catalog"]

# Create the DataCatalog instance from the configuration
Expand All @@ -372,7 +372,7 @@ Code in a Kedro project runs in one or more pipelines, where a pipeline is a ser
from kedro.config import OmegaConfigLoader
from kedro.io import DataCatalog

conf_loader = OmegaConfigLoader(".", base_env="", default_run_env="")
conf_loader = OmegaConfigLoader(conf_source=".")
conf_catalog = conf_loader["catalog"]
conf_params = conf_loader["parameters"]

Expand Down Expand Up @@ -546,7 +546,7 @@ And that's it. The notebook code has been refactored into a series of functions.
from kedro.config import OmegaConfigLoader
from kedro.io import DataCatalog

conf_loader = OmegaConfigLoader(".", base_env="", default_run_env="")
conf_loader = OmegaConfigLoader(conf_source=".")
conf_catalog = conf_loader["catalog"]
conf_params = conf_loader["parameters"]

Expand Down