Add a namespace for the bases and component

[asheeshgarg]

@DavidVujic
For the components and bases is it possible to add additional name spaces apart from global workspace namespace.
For example I am developing various connector for a project like http/ kafka etc I want to put them into bases called my_workpace/connector
-http

• kafka

I can create sub module inside the connector but when we run poly info it only show the top level component and also not tracked at the diff perspective in order to build it. Please suggest if I missed some other way to manage this

[DavidVujic]

(moving this into the discussions section)

[DavidVujic]

Hi @asheeshgarg! Thank you for your question. You are right, it is recommended to have a flat structure for what is called bricks (components and bases) in Polylith.

As you write, you can have sub-namespaces within a base or component, but it will be a part of the parent and not recognized as a separate brick. This is a design choice of the Polylith Architecture, and mainly because to make it simple to share and reuse code. As your example, the kafka namespace could be used in other places, if it is added as a separate component.

I hope this answers your question. I hope you will find the Polylith Architecture and the tooling support useful!

[saifmode]

Maybe I'm mistaken, but I experimented with a structure similar to OP's. I specified the bricks in one project as so:

"../../components/foo/connectors/connector_1.py" = "foo/connectors/connector_1.py"
"../../components/foo/connectors/connector_2.py" = "foo/connectors/connector_2.py"
"../../components/foo/connectors/connector_3.py" = "foo/connectors/connector_3.py"

I also had a connector_4.py that I didn't include. When I built the wheel and ran a docker container, I shelled onto the container and loaded a Python repl. From there I was able to import connectors 1 to 3, but not 4 (as expected).
So it seems you can treat bricks in this way. Is this not recommended? I thought it was quite useful as it means I can cut down on the number of dependencies for a project.

[DavidVujic]

Yes, that will work 😄

The wheel will be built with the modules that you have specified there. But this is including modules (i.e. python files), and not bricks (the namespace packages).

If you want to use poly tool to visualize and check, it won't recognize this kind of includes as bricks. Also, if one of the modules would reference the brick, based on what is defined in a __init__.py (which is also recommended, as a good practice) the code will fail during runtime. The added modules need to import the other ones with the full path to the module.

You can do it this way, but it's not the intended way and the tooling won't really be useful unfortunately.

[asheeshgarg]

Thanks @DavidVujic one follow up question for some of the data pipelines we have different implementation of components like Spark Vs Polars. Any recommendation do we have a single interface and implementation inside the components or we should create separate namespaces.

[DavidVujic]

I think it depends 😄 If the components are part of different projects, and contain different library dependencies, then maybe separate components is a good idea to not include any unused extras. But having them in one and the same component would work too.

[ybenitezf]

@asheeshgarg if boch the Spark and Polars implements the interface data pipeline, i think the better approach is literally to have a component describing the interface and then have to components implementing the interface with each technology, i.e.: if the namespace is my_org, i will do:

poetry poly add component --name=pipeline  # for the general interface
poetry poly add component --name=spark_pipeline  # spark implementation
poetry poly add component --name=polars_pipeline # polars implementation