How to deal with configuration files?

[Laurent-Zerah]

I've recently migrated my repo from a src layout to a Polilyth one, and I am quite satisfied with this decision. However, in the Polylith Architecture documentation, I did not find much about configuration files (or resource files, more generally) and the way to deal with them.

Previously, I had a directory named data which contained two files:

• a config.json file, in which I had everything related to constant variables, paths to useful resources, user settings, etc.;
• an __init__.py file, which loaded the config.json file, assigned it to a variable named cfg and added dynamically some more key/value pairs to it; afterward, cfg could be imported in any component related to my package.

Right now, I've reproduced this logic with a component named data_project_name which contains the same two files. But I find myself creating a new data_project_name with each new project I add to the repository, and it seems wrong.

What would be a better way ? Add the json configuration file in the project directory under "Projects" (but will it be added to the package when I build it?)? Use the [tool.*] table of the project's pyproject.toml? Put everything in the project's base directory and import from there? Or another approach?

[DavidVujic]

Hi @Laurent-Zerah, I'm happy to hear that you like Polylith! About resources, yes, maybe I should add suggestions about that in the docs. Good idea!

One option that I think of is to create one "config" component, and put the constants, paths and settings in there. They can be grouped by context, domain or someting that makes sense. If it's not a problem to share the component between the projects I would go for this approach. You could write the component with sub-namespaces, to make the separation clearer or just organize a data structure with keys and values.

Maybe some of the config values are loaded from the environment variables too? Then I think it would make a lot of sense to put it in one single component. To make components shareable, I would avoid (if possible) to dedicate them to a single project.

The way you have solved it today, is each project-named components loaded in bases for each specific projects?

If the configuration data should not be spread out in the different projects, I would try to refactor the json data into environment variables or secrets. Then the component(s) could be written in a more generic way.

[Laurent-Zerah]

Thanks David, sorry for the delay, I wanted to test things a little before getting back to you.

If it's not a problem to share the component between the projects I would go for this approach.

Indeed, but I fear It will be a bit messy in the long run.

Maybe some of the config values are loaded from the environment variables too?

That's the case, yes.

The way you have solved it today, is each project-named components loaded in bases for each specific projects?

Yes.

For now, here is how I refactored my monorepo:

• I created a new component named "configuration" which contains an __init__.py and a core.py with the following functions:

"""Interface for configuration component."""

import json
import pkgutil
from pathlib import Path
from typing import Any

from platformdirs import PlatformDirs, PlatformDirsABC


def load_config(source: str, filename: str) -> dict[str, Any]:
    """Load json config file.

    Args:
        source: location of configuration file.
        filename: configuration file name.

    Returns:
        Deserialized json file or empty dictionary.

    """
    if data := pkgutil.get_data(source, filename):
        return json.loads(data.decode(encoding="utf8"))
    return {}


def setup_ressource_directories(appname: str, appauthor: str) -> PlatformDirsABC:
    """Create directories where to place user data.

    Args:
        appname: name of the app.
        appauthor: name of the app author or distributing body for this application.

    Returns:
        platform directories.

    """
    dirs = PlatformDirs(appname=appname, appauthor=appauthor)
    Path(dirs.user_data_dir).mkdir(parents=True, exist_ok=True)
    Path(dirs.user_log_dir).mkdir(parents=True, exist_ok=True)
    return dirs

• then, each project has a dedicated subdirectory in configuration with its configuration.json file and the following kind of code in __init__.py:

"""Interface for project_1 configuration variables."""

from dev_python.configuration.core import load_config, setup_ressource_directories

cfg = load_config(__name__, "config.json")
dirs = setup_ressource_directories("project1", "MyProjects")

log_exceptions = dirs.user_log_dir + "/exceptions.log"
cfg["data"] = dirs.user_data_dir + "/data.json"

__all__ = ["cfg", "log_exceptions"]

• finally, I updated the repository pyproject.toml and the projects ones to include the new connfiguration component, and updated all imports.

Everything seems to work just fine, and the repository is much cleaner now, so it is a significant improvement, thank you very much.

One minor issue is that poly check reports 🤔 Cannot locate configuration in project1 if I set the pyproject.toml like this:

packages = [{include = "repo/configuration/project1", from = "../../components"}]

I have to set packages like this:

packages = [{include = "repo/configuration", from = "../../components"}]

Which means, if my understanding is correct, that each package will contain everything in configuration, which is not entirely satisfactory, but that will do anyway.

Thanks to @hstravis for his answer, which is certainly much more pythonic and robust, but it's a bit above what I'm able to do as a hobbyist without more detailed guidance.

[DavidVujic]

Yes, from a Polylith perspective, configuration is one component and it won't recognize importing a sub-namespace of the component directly.

[hstravis]

We had a similar problem. We recently switched from JSON config files to .env files. We use bash scripts to resolve config and generate .env files. We modified the component that previously loaded JSON to load from environment variables instead. For development, we configured VS Code to load the .env file. We use the same script and similar .env files for Kubernetes deployments.

scripts/
   app_foo/
      resolve_config.sh
   service_bar/
      resolve_config.sh
   components/
      result_config.sh

To quick switch between configurations, we have a simple bash function that calls the app-specific config:

switchconfig app_foo

We also added a pytest hook that hides tests if the corresponding config file is not available. Otherwise, we'd have thousands of tests loaded at the same time.