Thoughts about using uv workspace with polylith

[enadeau]

Hi,

I'm new to the polylith but I'm looking at using it in conjunction with uv to retructure a project. I'd like to use uv workspaces. The key benefit of it being to have a single shared lock file for all the projects. It ensures that that all the projects are running with the exact same version of the dependencies as development environment.

This seems to be a solution to #180

Is that a patterns that work well with polylith? This is not the way the polylith uv example does not use this feature and example with poetry seems to also use a different lockfile for each project.

I've at least been experimenting with it and seen the poly check missing some dependencies and not behaving as I'd expect it. Maybe this is related to having a unique lock file.

I'd be happy to provide and example or more detail if you think this is something worth exploring

[DavidVujic]

Hi @enadeau!

I have tried out uv workspaces and found that even if they share the same lock-file, the built artifacts are separate things. When building an app, that has the workspace dependencies to other parts, they will be represented as third-party deps when building wheels. I might have done things wrong, though. My impression is that workspaces has a similar developer experience as Polylith, with the editable dependencies.

I thought that I had solved most of the quirks with missing dependencies, but I probably have missed out on something. Yes, please share some more info about this so I can investigate! 🙏 The correctness of the poly check will be a lot better with a lock file. If the root lock file does not contain the all the dependencies, then I think a project-specific lock-file could solve it.

[enadeau]

Thanks for the quick response I've put a small example repo.

It contains a single base that uses fastapi and pydantic-settings. The corresponding project list fastapi as a dependencies but not pydantic-settings however poly check is not complaining.

I've also setup a specific branch with a lock file for the project but the poly check command is still not complaining

Let me know if I can provide any help

[DavidVujic]

This is great! I will use your example to find the issue. I have an idea where I might have done it wrong and with this I think I can find the problem (and hopefully a solution). Thank you!

[DavidVujic]

I think that I have located the problem, and have an idea for a solution.

Meanwhile, I think that adding --strict (uv run poly check --strict) will highlight the missing pydantic_settings library. I'll work on this during the week, because using strict shouldn't be needed in this case.

[enadeau]

Thanks for the response I've tested it on my example the strict flag works on the no-workspace branch but not the the main one

[DavidVujic]

I think that is because the main branch doesn't have a project-specific lock file. The Polylith tool will look into the root environment to figure out what the imports are (this, because the data in pyproject.toml is too sparse to get things right). The drawback is that it will do things wrong too. But the solution should be to add a project-specific lock file.

I am finishing up a new version that will default to strict mode when the dependencies were collected from lock files. I think that will solve the issue.

[DavidVujic]

There's a new version out (CLI v1.18.0, and Poetry plugin 1.30.0) 😄

It will use strict mode when the third-party dependencies are found in a lock file.

[enadeau]

Nice and thanks for quick response! I've updated the cli version on my example, and it now finds the missing dependency on the no-workspace branch.

For the main branch, there's a bit of a twist. When using workspaces, uv uses a single lock file for all the projects. So the lock file for the api project is located at the root of the repo.

This unique lock file ensures the same version of all dependencies are used across all projects.

You can actually get the locked dependencies for the api project when you run uv tree

$ uv tree
Resolved 31 packages in 29ms
api v0.1.0
├── fastapi v0.115.0
│   ├── pydantic v2.9.2
│   │   ├── annotated-types v0.7.0
│   │   ├── pydantic-core v2.23.4
│   │   │   └── typing-extensions v4.12.2
│   │   └── typing-extensions v4.12.2
│   ├── starlette v0.38.6
│   │   └── anyio v4.6.0
│   │       ├── idna v3.10
│   │       └── sniffio v1.3.1
│   └── typing-extensions v4.12.2
└── uvicorn v0.30.6
    ├── click v8.1.7
    ├── h11 v0.14.0
    ├── httptools v0.6.1 (extra: standard)
    ├── python-dotenv v1.0.1 (extra: standard)
    ├── pyyaml v6.0.2 (extra: standard)
    ├── uvloop v0.20.0 (extra: standard)
    ├── watchfiles v0.24.0 (extra: standard)
    │   └── anyio v4.6.0 (*)
    └── websockets v13.1 (extra: standard)
uv-workspace-polylith-example v0.1.0
├── fastapi v0.115.0 (*)
├── pydantic-settings v2.5.2
│   ├── pydantic v2.9.2 (*)
│   └── python-dotenv v1.0.1
├── uvicorn v0.30.6 (*)
├── fastapi-cli v0.0.5 (group: dev)
│   ├── typer v0.12.5
│   │   ├── click v8.1.7
│   │   ├── rich v13.8.1
│   │   │   ├── markdown-it-py v3.0.0
│   │   │   │   └── mdurl v0.1.2
│   │   │   └── pygments v2.18.0
│   │   ├── shellingham v1.5.4
│   │   └── typing-extensions v4.12.2
│   └── uvicorn[standard] v0.30.6 (*)
└── polylith-cli v1.18.0 (group: dev)
    ├── rich v13.8.1 (*)
    ├── tomlkit v0.13.2
    └── typer v0.12.5 (*)
(*) Package tree already displayed

Supporting this would be a nice featured to add to polylith. I'd be happy to help with that but would appreciate some pointer on where to get started 😃

[enadeau]

I think the best way would be to outsource the parsing to uv itself. A command like uv export produce quite a good result

$ uv export --package api --no-emit-project --no-hashes 
Resolved 31 packages in 4ms
# This file was autogenerated by uv via the following command:
#    uv export --package api --no-emit-project --no-hashes 
annotated-types==0.7.0
anyio==4.6.0
click==8.1.7
colorama==0.4.6 ; platform_system == 'Windows'
fastapi==0.115.0
h11==0.14.0
idna==3.10
pydantic==2.9.2
pydantic-core==2.23.4
sniffio==1.3.1
starlette==0.38.6
typing-extensions==4.12.2
uvicorn==0.30.6

I don't know that uv expose any api we could call from python and it will in general not be part of the virtual env so that migth not a really practical way.

I've also gotten a similar result with this simple parser I quickly put together. It is not perfect I know it at least ignore all optional dependencies.

import tomllib
import json
import pathlib


def get_member_packages_name(member_name: str, lock_file: pathlib.Path):
    with lock_file.open("rb") as fp:
        lock_file_data = tomllib.load(fp)
    assert member_name in lock_file_data["manifest"]["members"]
    packages = {package["name"]: package for package in lock_file_data["package"]}
    package_queue = [packages[member_name]]
    member_packages_name = set()
    while package_queue:
        package = package_queue.pop()
        member_packages_name.add(package["name"])
        all_dependecies = package.get("dependencies", [])
        package_queue.extend(
            packages[d["name"]]
            for d in all_dependecies
            if d["name"] not in member_packages_name
        )
    return member_packages_name


if __name__ == "__main__":
    lock_file = pathlib.Path("uv.lock")
    member_name = "api"
    res = get_member_packages_name(member_name, lock_file)
    for name in sorted(res):
        print(name)

I can maybe look a bit more into extracting the information from uv to avoid having to re-write the logic on our side

[DavidVujic]

Thank you for sharing ideas and code! ⭐

I've already started working on this and will try to incorporate it into the existing lock-file parsing, but passing a filtered view to the code that does that. Basically what you do in your example, very nice 👏

I'll make a pull request shortly, that I would like you to review it. I think that I have done something similar to what you do in the example (work in progress):

def get_dependencies(data: dict, package_name: str) -> list:
    res = next(p for p in data["package"] if p["name"] == package_name)

    dependency = [{res["name"]: res["version"]}]
    sub_dependencies = res.get("dependencies", [])
    nested_dependencies = [get_dependencies(data, p["name"]) for p in sub_dependencies]

    flattened_sub_dependencies = sum(nested_dependencies, [])

    if flattened_sub_dependencies:
        return dependency + flattened_sub_dependencies

    return dependency

(I'll probably return a toml structure instead, and pass that one into the existing flow)

[DavidVujic]

The work went smoother than I initially had thought, and I have released a new version with the support for uv workspaces as we have talked about here. The CLI version is 1.19.1. I have tested it on your example repo and think it works as expected.

I will announce this in the discussions forum shortly! Thank you again @enadeau for the ideas and suggetions for how to implement it, much appreciated 🙏

[enadeau]

Great, thanks for the new feature! I tested it on my example, and it reports the issue as expected 🎉

I have a real project at work that we are migrating to polylith and where I first encountered the issue. I'll give the new version a try on Monday and sees how it goes.

I might also find more idea for new feature as we move forward with the integration 😉

[DavidVujic]

Great! I found a bug when doing some more testing, and have released v1.19.2 fixing that. This will verify that the lock-file is enabled for workspace members, before parsing any dependencies.

[davzucky]

Thank you guys. I was looking precisely for that today. I just have to update poly-cli