Do not fail if conf/local is not present

[astrojuanlu]

Description

As per title.

Context

On a perfectly functioning project with an empty conf/local, if the directory is removed, kedro run will fail:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/juan_cano/.micromamba/envs/kpolars310/bin/kedro:8 in <module>                             │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/kedro/framework/cli/cl │
│ i.py:211 in main                                                                                 │
│                                                                                                  │
│   208 │   """                                                                                    │
│   209 │   _init_plugins()                                                                        │
│   210 │   cli_collection = KedroCLI(project_path=Path.cwd())                                     │
│ ❱ 211 │   cli_collection()                                                                       │
│   212                                                                                            │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/click/core.py:1130 in  │
│ __call__                                                                                         │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/kedro/framework/cli/cl │
│ i.py:139 in main                                                                                 │
│                                                                                                  │
│   136 │   │   )                                                                                  │
│   137 │   │                                                                                      │
│   138 │   │   try:                                                                               │
│ ❱ 139 │   │   │   super().main(                                                                  │
│   140 │   │   │   │   args=args,                                                                 │
│   141 │   │   │   │   prog_name=prog_name,                                                       │
│   142 │   │   │   │   complete_var=complete_var,                                                 │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/click/core.py:1055 in  │
│ main                                                                                             │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/click/core.py:1657 in  │
│ invoke                                                                                           │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/click/core.py:1404 in  │
│ invoke                                                                                           │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/click/core.py:760 in   │
│ invoke                                                                                           │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/kedro/framework/cli/pr │
│ oject.py:456 in run                                                                              │
│                                                                                                  │
│   453 │   node_names = node_names + nodes_names                                                  │
│   454 │   load_version = {**load_version, **load_versions}                                       │
│   455 │                                                                                          │
│ ❱ 456 │   with KedroSession.create(                                                              │
│   457 │   │   env=env, conf_source=conf_source, extra_params=params                              │
│   458 │   ) as session:                                                                          │
│   459 │   │   session.run(                                                                       │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/kedro/framework/sessio │
│ n/session.py:194 in create                                                                       │
│                                                                                                  │
│   191 │   │   session._store.update(session_data)                                                │
│   192 │   │                                                                                      │
│   193 │   │   # We need ConfigLoader and env to setup logging correctly                          │
│ ❱ 194 │   │   session._setup_logging()                                                           │
│   195 │   │   session_data.update(**_describe_git(session._project_path))                        │
│   196 │   │   session._store.update(session_data)                                                │
│   197                                                                                            │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/kedro/framework/sessio │
│ n/session.py:212 in _setup_logging                                                               │
│                                                                                                  │
│   209 │   def _setup_logging(self) -> None:                                                      │
│   210 │   │   """Register logging specified in logging directory."""                             │
│   211 │   │   try:                                                                               │
│ ❱ 212 │   │   │   logging_config = self._get_logging_config()                                    │
│   213 │   │   except MissingConfigException:                                                     │
│   214 │   │   │   self._logger.debug(                                                            │
│   215 │   │   │   │   "No project logging configuration loaded; "                                │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/kedro/framework/sessio │
│ n/session.py:201 in _get_logging_config                                                          │
│                                                                                                  │
│   198 │   │   return session                                                                     │
│   199 │                                                                                          │
│   200 │   def _get_logging_config(self) -> dict[str, Any]:                                       │
│ ❱ 201 │   │   logging_config = self._get_config_loader()["logging"]                              │
│   202 │   │   # turn relative paths in logging config into absolute path                         │
│   203 │   │   # before initialising loggers                                                      │
│   204 │   │   logging_config = _convert_paths_to_absolute_posix(                                 │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/kedro/config/templated │
│ _config.py:154 in __getitem__                                                                    │
│                                                                                                  │
│   151 │   │   # explicitly on the ``TemplatedConfigLoader`` instance.                            │
│   152 │   │   if key in self:                                                                    │
│   153 │   │   │   return super().__getitem__(key)                                                │
│ ❱ 154 │   │   return self.get(*self.config_patterns[key])                                        │
│   155 │                                                                                          │
│   156 │   def __repr__(self):  # pragma: no cover                                                │
│   157 │   │   return (                                                                           │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/kedro/config/templated │
│ _config.py:186 in get                                                                            │
│                                                                                                  │
│   183 │   │   Raises:                                                                            │
│   184 │   │   │   ValueError: malformed config found.                                            │
│   185 │   │   """                                                                                │
│ ❱ 186 │   │   config_raw = _get_config_from_patterns(                                            │
│   187 │   │   │   conf_paths=self.conf_paths, patterns=patterns, ac_template=True                │
│   188 │   │   )                                                                                  │
│   189 │   │   return _format_object(config_raw, self._config_mapping)                            │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/kedro/config/common.py │
│ :73 in _get_config_from_patterns                                                                 │
│                                                                                                  │
│    70 │                                                                                          │
│    71 │   for conf_path in conf_paths:                                                           │
│    72 │   │   if not Path(conf_path).is_dir():                                                   │
│ ❱  73 │   │   │   raise ValueError(                                                              │
│    74 │   │   │   │   f"Given configuration path either does not exist "                         │
│    75 │   │   │   │   f"or is not a valid directory: {conf_path}"                                │
│    76 │   │   │   )                                                                              │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Given configuration path either does not exist or is not a valid directory: /Users/juan_cano/Projects/QuantumBlack 
Labs/workshop-jupyter-kedro/conf/local

Just recreating the directory mkdir conf/local makes everything work again.

When converting an existing project to a Kedro one (gh-410), this creates unnecessary toil, because it forces the user to create some directories and files even though initially they are empty or unused.

Notice that this doesn't mean removing conf/local from the project templates - just being more permissive if users are not starting with kedro new.

Possible Implementation

Issuing a warning if the directory is not found, and proceeding as normal.

$ kedro run
[05/20/23 11:07:19] INFO     Kedro project workshop-jupyter-kedro                                                                            session.py:359
[05/20/23 11:07:19] WARNING  Configuration path for default overriding environment `local` not found, proceeding without overrides
[05/20/23 11:07:22] INFO     Loading data from 'openrepair-0_3-events-raw' (CSVDataSet)...                                              data_catalog.py:345
...

If the environment is manually set, the current error could be kept (but not the traceback, see gh-2401):

$ kedro run --env=server
[03/08/23 11:43:36] INFO     Kedro project test-omegaconf                                                                                session.py:355
[03/08/23 11:43:36] ERROR  Configuration path for manually set environment `server` not found, path either does not exist or is not a valid directory: /Users/juan_cano/Projects/QuantumBlack Labs/workshop-jupyter-kedro/conf/server
$ echo $?
1

Possible Alternatives

Alternatively, we could keep the error for all cases, but conceal the traceback as argued in gh-2401.

[merelcht]

Double check with @idanov why this is so strictly enforced.

[jmholzer]

+1, would love to see this change.

[merelcht]

I talked to @idanov about this and he said the main reason this error has been put in is to make users aware that config is layered: as in it's not just the one file, but you can have different environments to override whatever is in base. If the local directory wouldn't be there, it would be much harder for users to discover the environment functionality.

I see the benefit in keeping to error to create awareness. Has anyone seen complaints from users about this behaviour?

[astrojuanlu]

That makes sense.

I personally haven't seen any complaints, because I assume most people run kedro new, which brings an empty conf/local directory.

For anyone trying to manually craft a Kedro project in an existing directory (without doing kedro new in a separate directory and in the absence of a kedro init command #2512) it's an extra directory to create for nothing.

At the minimum, if we are to keep the error message (which I'm still not sold on), I think we should conceal the traceback of this one #2401

[astrojuanlu]

We have now alternative ways to use Kedro as a library without submitting to all the assumptions of Kedro framework, so given that there was support to keep the error message and that we have already agreed to do #2971, I'm closing this issue as not planned.

[astrojuanlu]

...but let's try to make the traceback more manageable #2401

[astrojuanlu]

FTR, I'm personally struggling again with this on Databricks...