Addons implementation

[pvk-developer]

Expected behavior

We want to create different families of transformers and those to be optionally installed.

Aproach

Using Namespace Packaging we will be able to achieve multiple packages within the RDT project and those to be installed afterwards as addons and only be installed if the user actually wants them.

Structure

The addons folder for the multiple families should be created within the rdt/transformers/ without a __init__.py file inside it. This folder will contain multiple folders representing a family which will become a package.

rdt
├── hyper_transformer.py
├── __init__.py
└── transformers
    ├── addons
    │   ├── digital
    │   ├── geo
    │   ├── time
    │   └── web
    ├── base.py
    ├── boolean.py
    ├── categorical.py
    ├── __init__.py
    ├── null.py
    └── numerical.py

Each family folder will contain a __init__.py file, a config.json file and the python modules that contain code for this addon.

addons/time/
├── config.json
├── datetime.py
└── __init__.py

Config file (JSON)

The config file for each addon has the following structure:

• name: The name of the package / library. This is usually rdt_<family_name>. Required.
• transformers: A list of transformers that can be imported from this family. Required
• requirements: A list of requirements for that package. Only required if there are any requirements needed to be installed.

{
    "name": "rdt_time",
    "requirements": [
        "maya"
    ]
    "transformers": [
        "rdt.transformers.addons.time.datetime.DatetimeTransformer",
        "rdt.transformers.addons.time.datetime.SplitDatetimeTransformer",
        "rdt.transformers.addons.time.dummy.DummyTimeTransformer"
    ]
}

Changes to the current code

setup.py

The setup.py file has to be adapted to:

• Read all the yaml dependencies so those can be installed for testing environment.
• Use find_namespace_package instead of find_packages, and specify what to be found (change the rdt.* for rdt.transformers ).
• Add yaml dependency.

rdt/transformers/__init__.py

The __init__.py must dynamically detect and import the installed addons. This is done by reading if found any config.yaml file on the addons/ folders.

The best aproach for this is by using importlib.

Multiple Package Deployment

A dynamic way of deploying the packages to PyPI by using the config.json file for each family must be created for easy maintanance and release process.