How to build & run a container image with docker-compose.yaml

[extreme4all]

python-polylith-example-uv/projects/my_fastapi_project/README.md

Lines 3 to 20 in ac4e868

	## Build the project
	Navigate to this folder (where the `pyproject.toml` file is)
	1. Export the dependencies (when using uv workspaces and having no project-specific lock-file):
	``` shell
	uv export --no-emit-project --output-file requirements.txt
	```
	2. Build a wheel:
	``` shell
	uv build --out-dir ./dist
	```
	## Build a docker image
	``` shell
	docker build -t myimage .
	```

I have been following the provided steps to build and run the project, which involve:

Exporting dependencies using uv export to generate requirements.txt.
Building a wheel file via uv build.
Building a Docker image using docker build.

These steps work well, but I’m looking for a more practical and efficient way to manage the project lifecycle, particularly when iterating on development. Instead of building the project each time, I’d like to simplify the process, ideally leveraging docker-compose to streamline dependencies and project setup.

as extra context, this is the project i'm working on https://github.com/Bot-detector/bot_detector

[DavidVujic]

For local development I would recommend to run code and tests from the top of the repo: the Polylith workspace. The virtual environment there should have all the necessary dependencies and paths setup already from the top-level pyproject. I usually develop that way, running code in a REPL and develop & evaluate code all the time (not building wheels at this stage). Would that work for you?

When deploying (to a local cluster or to a remote environment): that's where building wheels and docker containers would fit in the workflow.

[extreme4all]

for local development i do extensively use docker containers, i'm kinda not getting why we even need a wheel if i'm just deploying an api, and if we are deploying from the root (top of the repo) than what is the purpose of the /projects.

i do need a place for a Dockerfile, so that would be /projects/ i suppose and than i just copy all the bases, components etc over?

edit;
i guess i can make one Dockerfile with all bases, components and specify the entrypoint in the docker-compose.yaml, and deploy that towards my cluster also, is that the intended way?

[DavidVujic]

The wheel is basically a way to collect all the needed bricks (components and bases) for a specific project and the needed third-party dependencies. But I think the same thing can be achieved with uv sync maybe?

I don't think you should deploy code from the root. That's what the projects are for. In a Monorepo, you will have several bricks, but each project will likely only need a subset of those. The top-level contains all the bricks, and is there for development: to have all the code available for development, testing and experimentation.

I think a dockerfile for a specific project would fit well to live in the projects folder. If you need something to run your development flow in, then maybe a docker/compose file at the top-level would make sense. I usually run code on the local machine without docker (unless it is a dependent third-party service such as Kafka or similar, those ones I run in docker and "my" code locally).

I either the functions directly in a REPL or if I want to run a FastAPI app (or similar), then I execute the code locally too (the "app" parts and endpoints in a base).

[extreme4all]

I don't think you should deploy code from the root. That's what the projects are for.

i agree

I think a dockerfile for a specific project would fit well to live in the projects folder.

i agree

I either the functions directly in a REPL or if I want to run a FastAPI app (or similar), then I execute the code locally too (the "app" parts and endpoints in a base).

i would like to be able to run it in the terminal but also with docker-compose, the idea is that i can spin up multiple services at once locally. the challenge is that i can't get it to work even with the building, i'm a bit lost here cause i don't really understand the building process and the purpose, i only ever did it for a package and there it magically works when someone imports the package from pypi but in my container i just seem to have the wheel & requirements.txt

[DavidVujic]

It would be great to have a smooth solution to that! I haven't that much experience of local development in that kind of style, other than deploying individual services into a local K8s cluster by packaging and installing them from built wheels. Something like the Dockerfile here: https://github.com/DavidVujic/python-polylith-example-uv/blob/ac4e86849d4be010daa1519bd87289630fa4ff7b/projects/my_fastapi_project/Dockerfile

With this, one would probably want a script like make deploy or something similar, that will run uv build for all the projects in the repo (or a selection o them) and also build & deploy the docker containers (maybe using Docker Compose).

To deploy one service from the monorepo, I would recommend to build a wheel and then install that wheel in the Docker container. I know others copy the necessary folders from the repo into a container and then run uv sync. I'll try to find an example - there's one in an github issue somewhere 😄

[DavidVujic]

I think I found an alternative to building a wheel and install it in a container, from this resolved issue (the first comment), I think there are a bunch of optimization flags there that might not be relevant to your setup. But they run uv sync for an individual project here: DavidVujic/python-polylith#307

My preference would still be wheels + pip install though.

[extreme4all]

It would be great to have a smooth solution to that! I haven't that much experience of local development in that kind of style, other than deploying individual services into a local K8s cluster by packaging and installing them from built wheels. Something like the Dockerfile here: https://github.com/DavidVujic/python-polylith-example-uv/blob/ac4e86849d4be010daa1519bd87289630fa4ff7b/projects/my_fastapi_project/Dockerfile

With this, one would probably want a script like make deploy or something similar, that will run uv build for all the projects in the repo (or a selection o them) and also build & deploy the docker containers (maybe using Docker Compose).

To deploy one service from the monorepo, I would recommend to build a wheel and then install that wheel in the Docker container. I know others copy the necessary folders from the repo into a container and then run uv sync. I'll try to find an example - there's one in an github issue somewhere 😄

i'm still pretty unclear on how to run it, you can see the my error

i managed to build the wheel in docker

# projects/hiscore_scraper/Dockerfile
# Build stage: Use uv to build the wheel
FROM python:3.11-slim-bookworm AS builder

# Copy uv from external repository
COPY --from=ghcr.io/astral-sh/uv:0.5.13 /uv /uvx /bin/

# Set the working directory for the build stage
WORKDIR /app

# Copy only necessary files for building the wheel
COPY ./projects/hiscore_scraper/pyproject.toml .
COPY ./projects/hiscore_scraper/uv.lock .
COPY ./projects/hiscore_scraper/ .

# Build the wheel using uv
RUN uv build --out-dir ./dist

# Production stage: Prepare the final production environment
FROM python:3.11-slim-bookworm AS production

# Copy uv from the builder stage
COPY --from=ghcr.io/astral-sh/uv:0.5.13 /uv /uvx /bin/

# Copy the built wheel from the build stage
WORKDIR /app
COPY --from=builder /app/dist/ .
COPY --from=builder /app/pyproject.toml .
COPY --from=builder /app/uv.lock .
# Install the wheel using uv
# RUN uv install --wheel ./dist/*.whl

# Run the package using uv
CMD ["uv", "run", "hiscore_scraper"]

[build-system]
requires = ["hatchling", "hatch-polylith-bricks"]
build-backend = "hatchling.build"

[project]
name = "hiscore_scraper"
version = "0.1.0"

requires-python = ">=3.10"

dependencies = [
    "aiokafkaengine>=0.1.0",
    "osrs>=0.0.14",
    "pydantic-settings>=2.7.0",
    "pydantic>=2.10.4",
]

[project.scripts]
hiscore_scraper = "bot_detector.hiscore_scraper.core:main"

# This section is needed for building
[tool.hatch.build.targets.wheel]
packages = ["bot_detector"]

[tool.hatch.build.hooks.polylith-bricks]

[tool.polylith.bricks]
"../../bases/bot_detector/hiscore_scraper" = "bot_detector/hiscore_scraper"
"../../components/bot_detector/proxy_manager" = "bot_detector/proxy_manager"
"../../components/bot_detector/schema" = "bot_detector/schema"

to run it from the root in my repo https://github.com/Bot-detector/bot_detector

docker build -f "./projects/hiscore_scraper/Dockerfile" -t hiscore_scraper . && docker run -d --name hiscore_scraper_debug hiscore_scraper sleep infinity && docker exec -it hiscore_scraper_debug bash

$ uv run hiscore_scraper
Traceback (most recent call last):
  File "/app/.venv/bin/hiscore_scraper", line 5, in <module>
    from bot_detector.hiscore_scraper.core import main
ModuleNotFoundError: No module named 'bot_detector'

root@51d77b1256b6:/app/hiscore_scraper-0.1.0# tree
.
├── Dockerfile
├── PKG-INFO
├── bot_detector
│   ├── hiscore_scraper
│   │   ├── __init__.py
│   │   └── core.py
│   ├── proxy_manager
│   │   ├── __init__.py
│   │   └── core.py
│   └── schema
│       ├── __init__.py
│       └── core.py
├── pyproject.toml
├── requirements.txt
└── uv.lock

[DavidVujic]

I'm not sure if I understand it correctly, but shouldn't the first part of the dockerfile copy the entire repo structure, if it should be able to create a wheel from Python code in the top-level components and bases?

The second part of the dockerfile: I don't think you anything else than the wheel itself (and a requirements.txt if you want to do a 2-step install with fixed third-party dependencies from a lock file). I don't think you need uv here either (only needed to build the wheel). You can install it in the docker containing with pip install. When installed, I think you should be able to run the script directly, because it is added in "globally" within the container.

[extreme4all]

the tree you see is from within the dockerfile, its the created wheel so it has the files but when i try to run it than it says module not found, because i import from bot_detector (in your case from example) but it somehow fails to see "bot_detector" as a package after the building

[extreme4all]

you know what maybe you are right just double checking stuf

[extreme4all]

okay still alot of work in progress going on here but i got the following

FROM python:3.11-slim-bookworm AS builder

# Copy uv from external repository
COPY --from=ghcr.io/astral-sh/uv:0.5.13 /uv /uvx /bin/

# Set the working directory for the build stage
WORKDIR /app

# Copy only necessary files to run the projects
COPY ./bases ./bases
COPY ./components ./components
COPY ./projects/hiscore_scraper .

RUN uv sync --frozen --no-editable
RUN uv build --out-dir ./dist
RUN uv pip install dist/hiscore_scraper-0.1.0-py3-none-any.whl
RUN uv sync --frozen --no-editable

# Production stage: Prepare the final production environment
FROM python:3.11-slim-bookworm AS production
# Copy uv from external repository
COPY --from=ghcr.io/astral-sh/uv:0.5.13 /uv /uvx /bin/
COPY --from=builder --chown=app:app /app/.venv /app/.venv

WORKDIR /app

CMD ["uv","run","hiscore_scraper"]

i can validate that hiscore_scraper is succesfully build inside the container

root@722dd6c3e610:/app# cat .venv/bin/hiscore_scraper 
#!/app/.venv/bin/python3
# -*- coding: utf-8 -*-
import re
import sys
from bot_detector.hiscore_scraper.core import run
if __name__ == "__main__":
    sys.argv[0] = re.sub(r"(-script\.pyw|\.exe)?$", "", sys.argv[0])
    sys.exit(run())

however now bot_detector is not in the container packages

root@722dd6c3e610:/app/.venv/lib/python3.11/site-packages# ls
AioKafkaEngine                    annotated_types-0.7.0.dist-info     frozenlist-1.5.0.dist-info       pydantic-2.10.4.dist-info
__pycache__                       async_timeout                       hiscore_scraper-0.1.0.dist-info  pydantic_core
_virtualenv.pth                   async_timeout-5.0.1.dist-info       idna                             pydantic_core-2.27.2.dist-info
_virtualenv.py                    asyncio                             idna-3.10.dist-info              pydantic_settings
aiohappyeyeballs                  asyncio-3.4.3.dist-info             multidict                        pydantic_settings-2.7.0.dist-info
aiohappyeyeballs-2.4.4.dist-info  attr                                multidict-6.1.0.dist-info        python_dotenv-1.0.1.dist-info
aiohttp                           attrs                               multidict.libs                   requests
aiohttp-3.11.11.dist-info         attrs-24.3.0.dist-info              osrs                             requests-2.32.3.dist-info
aiokafka                          certifi                             osrs-0.0.14.dist-info            typing_extensions-4.12.2.dist-info
aiokafka-0.12.0.dist-info         certifi-2024.12.14.dist-info        packaging                        typing_extensions.py
aiokafkaengine-0.1.0.dist-info    charset_normalizer                  packaging-24.2.dist-info         urllib3
aiosignal                         charset_normalizer-3.4.1.dist-info  propcache                        urllib3-2.3.0.dist-info
aiosignal-1.3.2.dist-info         dotenv                              propcache-0.2.1.dist-info        yarl
annotated_types                   frozenlist                          pydantic                         yarl-1.18.3.dist-info

[extreme4all]

huray, i got it working

python really didn't like that i ran in the /app instead of the /app/projects/hiscore_scraper

FROM python:3.11-slim-bookworm AS builder

# Copy uv from external repository
COPY --from=ghcr.io/astral-sh/uv:0.5.13 /uv /uvx /bin/

# Set the working directory for the build stage
WORKDIR /app

# Copy only necessary files to run the projects
COPY ./bases ./bases
COPY ./components ./components
COPY ./projects ./projects

WORKDIR /app/projects/hiscore_scraper

RUN uv venv
RUN uv build --out-dir ./dist
RUN uv pip install /app/projects/hiscore_scraper/dist/hiscore_scraper-0.1.0-py3-none-any.whl
RUN uv sync --frozen --no-editable

# Production stage: Prepare the final production environment
FROM python:3.11-slim-bookworm AS production

WORKDIR /app/projects/hiscore_scraper

COPY --from=builder --chown=app:app /app/projects/hiscore_scraper/.venv /app/projects/hiscore_scraper/.venv

CMD [".venv/bin/hiscore_scraper"]

[extreme4all]

i think this is a lot cleaner having the build in the container itself, and a generic Dockerfile in the root of the repository that mounts the bases, components, projects so you can run exec into the container and run from there, which allows networking to other containers similarly as you'd have in production

here is my repository
https://github.com/Bot-detector/bot_detector

[extreme4all]

maybe this is useful for the example repository?

[DavidVujic]

Very nice! 👏

Do you need to do build, pip install and uv sync here?

RUN uv venv
RUN uv build --out-dir ./dist
RUN uv pip install /app/projects/hiscore_scraper/dist/hiscore_scraper-0.1.0-py3-none-any.whl
RUN uv sync --frozen --no-editable

I'm thinking it should work either by building + pip installing, or uv sync. Both options will add the code to the virtual environment, and the venv is copied into the second docker step (as I understand the Dockerfile).

Alternatively, you can build a wheel, and copy that one into the next step where you do a pip install.