Optimize Dockerfile for Layer Caching

[MickaelCa]

Problem: Unable to Leverage Docker Layer Caching for Dependencies

What we want to achieve

We aim to speed up Docker builds by installing dependencies before copying the project source code. This enables Docker to cache the dependency layer, avoiding unnecessary reinstallation when only code changes.

Typical approach example:

COPY pyproject.toml .
RUN pip install ...
COPY src/ ./src/

---

Current issue

• Our project uses:

  • A src/ layout
  • pyproject.toml with pip install . for installation

• This structure requires the source code to be present at install time.

• Installing dependencies before copying src/ fails with errors such as:

error in 'egg_base' option: 'src' does not exist or is not a directory

➡️ Result: We cannot leverage Docker layer caching in the standard way because the install step depends on having the code available.

---

Possible solutions

1. Maintain a separate requirements.txt
   Install dependencies from it before copying code.
   🔴 Downside: Requires manually keeping dependencies in sync.

2. Use a tool supporting dependency-only installs
   For example, with Poetry:

   poetry install --no-root

   or similar with Hatch, to install dependencies without building the package.

3. Accept the limitation
   Document that Docker caching is not feasible with the current setup due to packaging constraints.

4. Refactor project structure
   Explore decoupling dependency installation from package building, if feasible.