Aliases configuration file

[Peder2911]

Is your feature request related to a problem? Please describe.
The current way of providing aliases via the --alias flag does not provide any reliable way to share or replicate aliases between team members.

Describe the solution you'd like
My current workaround is to maintain the aliases in a file inside of the repo (.aliases). It would be good if the tool supported this natively.

Describe alternatives you've considered
I feel like it would be a good idea if the poly CLI either could accept a file containing newline-separated aliases directly, or if it had a configuration setting in a TOML file somewhere where one could set the aliases. I would like to hear your opinion on whether this configuration should happen in a separate file (i.e $USER/.poly.toml or $(pwd)/poly.toml) or whether it should be located in the development project pyproject file.

Hope you like my idea! I could implement this and PR, i would just need to hear your preference between the above alternatives.
P

[Peder2911]

To give you some context, I am currently using this file (.aliases)

pyyaml=yaml
opentelemetry-instrumentation=opentelemetry
opentelemetry-instrumentation-fastapi=opentelemetry
opentelemetry-instrumentation-aiohttp-client=opentelemetry
opentelemetry-distro[otlp]=opentelemetry
opentelemetry-instrumentation=opentelemetry
pydantic-settings=pydantic_settings
azure-identity=azure
pydantic[email]=pydantic

and this command when running check:

rye run poly check --alias $(cat .aliases | tr '\n' ','| sed 's/,$//') --strict

[DavidVujic]

I like your ideas @Peder2911!

I think it would make a lot of sense to have that in a TOML file, in a similar way as when configuring other tools (such as mypy).

Some of the aliases in your list I would expect the tool to figure out without it, I will have a look at this.

Edit: probably because of the --strict option.

[DavidVujic]

Meanwhile, would it be an option to use the Rye [tool.rye.scripts] section to define custom scripts, that also would be versioned?

[Peder2911]

Oh cool, I wasn't aware of rye scripts. That's a good solution!
I was thinking about whether the configuration for Polylith should be a part of the pyproject.toml at the root, or whether it should live in its own file. Mypy of course has a pretty excellent config resolution that looks in multiple places, that would be nice tool.

[svartalf]

I have a similar problem with the Google Cloud libraries specifically, though there are more examples out there in the PyPI.

When we add a couple of the Google libraries to the dependencies list,

dependencies = [
    "google-cloud-storage",
    "google-cloud-bigquery",
]

we will need to use them like this, according to their examples and docs:

from google.cloud import storage
from google.cloud import bigquery

bigquery_client = bigquery.Client()
storage_client = storage.Client()

In the current Polylith state, it would cause an issue with the poly check or poly libs commands, as they'll fail with this error:

$ poly check
🤔 Cannot locate google in my_project

Introspecting the dependencies metadata with the importlib.metadata to see what packages are providing this namespace might also be an option here, e.g.:

>>> from importlib.metadata import packages_distributions
>>> packages_distributions()["google"]
['google-api-core', 'protobuf', 'google-cloud-bigquery-storage', 'google-resumable-media', 'google-cloud-storage', 'google-auth', 'google-cloud-core', 'google-cloud-bigquery', 'googleapis-common-protos']

The downside of this approach is that I can use the google-cloud-storage client in my source code, but add the google-cloud-bigquery dependency, which will make this check happy, but also incorrect.

[DavidVujic]

Maybe using the rye feature is a good-enough solution, what do you think?

(I haven't tried this out properly, though)

[tool.rye.scripts]
check = { cmd = [
      "poly",
      "check",
      "--alias", "pyyaml=yaml,opentelemetry-instrumentation=opentelemetry, opentelemetry-instrumentation-fastapi=opentelemetry, opentelemetry-instrumentation-aiohttp-client=opentelemetry, opentelemetry-distro[otlp]=opentelemetry, opentelemetry-instrumentation=opentelemetry, pydantic-settings=pydantic_settings, azure-identity=azure, pydantic[email]=pydantic",
      "--strict"]}

[DavidVujic]

Great input @svartalf! I will try this out using the Google packages and try to find a solution.

[DavidVujic]

@svartalf I just tried with the google libraries and the check command shouldn't notify about the missing google library, if it is added in the project. However, I also found that it will report a false-positive when using bigquery but when only installed storage. This is a problem that I will investigate further, currently the tool takes some shortcuts to figure out the libraries part.

[DavidVujic]

@Peder2911 If you run the check command without the --strict option, I think you will have less need to specify aliases. The tradeoff is that the check command will report on false-positives for libraries like opentelemetry and google, that uses the same top namespace for several different installable packages.

Otherwise, I would suggest to use the Rye script (see suggestion).

[DavidVujic]

Oh cool, I wasn't aware of rye scripts. That's a good solution! I was thinking about whether the configuration for Polylith should be a part of the pyproject.toml at the root, or whether it should live in its own file. Mypy of course has a pretty excellent config resolution that looks in multiple places, that would be nice tool.

Oh, sorry. I must have missed this one. If you want to add something within Polylith, I think that the pyproject.toml at the root might be a good option! I also think that the already existing rye scripts option could be a candidate for solving this issue.

I'm not happy with how the check command is working as of today though, with the need to add the aliases for libraries like opentelemetry. I will spend some time to figure out a possible better solution (the one today takes some shortcuts to locate dependencies).

[DavidVujic]

@svartalf Will using the --alias option for poly check and poly libs solve it for you?

I am guessing that you use the --strict option already? So that means the tool expect an almost extact naming match. To solve it for libraries like the ones provide by google you can use --alias.

Example

# check
rye run poly check --alias google-cloud-storage=google google-cloud-bigquery=google --strict

#libs
rye run poly libs --alias google-cloud-storage=google google-cloud-bigquery=google --strict

Running the commands without the --strict option you won't be needing the alias option either. That will result in possible false-positive as you described, reporting "ok" for a google-cloud-* that is not added in the project.

[DavidVujic]

(moving this to the 💡 discussions section)

[DavidVujic]

Update: there's a new version of the Polylith CLI, that I believe solves the issues with locating the third-party dependencies that is put in sub-namespaces of a global one, such as opentelemetry and google.

This would also remove the need to add the --alias option. Here's a recording where I add the opentelemetry-instrumentation-fastapi library and getting recognized when running the poly check command.

I hope this new version will solve your issues, if they still are relevant, @Peder2911 and @svartalf!

I would also appreciate if you have the time to test this out and share your feedback 🙏

EDIT:
(This could open up for possible false-positives, when using several libraries using the same top namespaces. Not sure, I need to test that out, and appreciate help with that part)

EDIT:
Here's the pull request: #219

[DavidVujic]

Ongoing work to make the dependency lookup more reliable (parsing project-specific lock files): https://github.com/DavidVujic/python-polylith/pull/220/files

[shaneseaton]

I am coming across this issue with the python-jose package.

[tool.poetry.dependencies]
python-jose = {extras = ["cryptography"], version = "^3.3.0"}

import jose

Check fails to pick it up (I am on poetry-polylith-plugin 1.23.0). Having the aliases as a list somewhere would be nice so they can be applied to all commands. The scripts idea is ok, but having to keep up to date for all check, info, libs etc is not great.

Could there be a place for this in say the workspace.toml

[DavidVujic]

I think this specific thing is a bug in Polylith, it should pick up the jose top-level name from the distribution data. I probably have expected a different format than what's in there.

[DavidVujic]

Issue added: #229

[DavidVujic]

@shaneseaton I think this specific problem with the python-jose import is solved with the latest release of the Poetry plugin. When the dependency is added to the project (and the project-specific lock file is updated) it should recognize the import jose as coming from python-jose.

[hstravis]

I ran into this with pymongo and bson. The bson package is installed when pymongo is installed - not as a dependency but as an adjacent package. It is clear to see in the pymongo repo. Would be nice if I could set the alias in pyproject.toml or poetry.toml. I'm using poetry-polylith-plugin v1.24.0 and poetry v1.8.3.

[DavidVujic]

Thank you for sharing this! This way of including packages was unknown to me. I would expect bson to be in the lock-file of a project, but it is included in a different way I guess. Otherwise, the Polylith tool would have recognized it as a third-party subdependency.

[DavidVujic]

@hstravis There's a new version of the Poetry plugin (v1.26.2) that should recognize packages like bson when included as an adjacent package. Implementation details in https://github.com/DavidVujic/python-polylith/releases/tag/stable-39

Please let me know if this solves the issue with the pymongo and bson package!

[hstravis]

@DavidVujic Yep that works. Thanks!

[devicieRK]

Not sure if this should be working without an alias (alias is workable ATM)
I get this issue with pyjwt (import jwt)
Using PDM and polylith-cli │ 1.12.0

[devicieRK]

Apologies, yes I didn't mention that. We use strict for the new 3rd party version checks.
It would be good to either have a separate strict flag for these features or as mentioned the alias in pyprojet.toml workspace.toml etc

[DavidVujic]

This specific thing with pyjwt is a bug, because it should recognize it also with the --strict flag without any aliases. I will fix this and release a new version shortly. Thank you for reporting this! ⭐

[DavidVujic]

I have published a new version of the CLI and the Poetry plugin, that should solve this specific case. Please let me know it it solves the issue.

[DavidVujic]

I'll consider adding the configurable option (I'm a bit hesitant though, would rather try making the command as correct as possible without aliases).

Meanwhile, PDM also support writing scripts in the pyproject.toml file. Would adding something there, using the poly command with the aliases, be an alternative?

[devicieRK]

Thanks @DavidVujic that solves it.

PDM Scripts might help I'll take a look, we have a rather complex multi language repo with other script and task managers so it might not be as straight forward, but always good to have options