Skip to content

Latest commit

 

History

History
52 lines (35 loc) · 2.79 KB

credentials.md

File metadata and controls

52 lines (35 loc) · 2.79 KB
Raw

Credentials

For security reasons, we strongly recommend that you do not commit any credentials or other secrets to version control. Kedro is set up so that, by default, if a file inside the conf folder (and its subfolders) contains credentials in its name, it will be ignored by git.

Credentials configuration can be used on its own directly in code or fed into the DataCatalog. If you would rather store your credentials in environment variables instead of a file, you can use the OmegaConfigLoader to load credentials from environment variables as described in the advanced configuration chapter.

How to load credentials in code

Credentials configuration can be loaded the same way as any other project configuration using any of the configuration loader classes: ConfigLoader, TemplatedConfigLoader, and OmegaConfigLoader.

The following examples are valid for both, the ConfigLoader and the OmegaConfigLoader.

from pathlib import Path

from kedro.config import OmegaConfigLoader
from kedro.framework.project import settings

# Substitute <project_root> with the [root folder for your project](https://docs.kedro.org/en/stable/tutorial/spaceflights_tutorial.html#terminology)
conf_path = str(Path(<project_root>) / settings.CONF_SOURCE)
conf_loader = ConfigLoader(conf_source=conf_path)
credentials = conf_loader["credentials"]

This loads configuration files from conf/base and conf/local whose filenames start with credentials, or that are located inside a folder with a name that starts with credentials.

Calling conf_loader[key] in the example above throws a MissingConfigException error if no configuration files match the given key. But if this is a valid workflow for your application, you can handle it as follows:

from pathlib import Path

from kedro.config import OmegaConfigLoader, MissingConfigException
from kedro.framework.project import settings

conf_path = str(Path(<project_root>) / settings.CONF_SOURCE)
conf_loader = OmegaConfigLoader(conf_source=conf_path)

try:
    credentials = conf_loader["credentials"]
except MissingConfigException:
    credentials = {}
The `kedro.framework.context.KedroContext` class uses the approach above to load project credentials.

How to work with AWS credentials

When you work with AWS credentials on datasets, you are not required to store AWS credentials in the project configuration files. Instead, you can specify them using environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and, optionally, AWS_SESSION_TOKEN. Please refer to the official documentation for more details.