Hardlink-based install mode for shared package storage across venvs

[polfeliu]

Hi everyone,

Recently I've been using git worktrees more and more for development, and it's greatly improved my workflow by cutting down on context switching overhead. Around the same time, I had the bad luck of an old SSD dying on me, apparently from exceeding its write rating (>100TB). That led me to investigate what's generating the most writes on my machine and CI servers.

Virtualenvs across worktrees and CI turned out to be a significant part of that, with some projects having >1GB venvs being recreated often. Some dependencies in my project change very often while others change rarely, so using a single shared venv across worktrees isn't reliable, each worktree needs its own environment that reflects its own lockfile. Poetry's cache avoids re fetching packages from the index, but it doesn't prevent environments from being recreated from scratch each time, which still means writing all those files to disk again.

That led me to experiment with a hardlink based install mode: wheels get unpacked once into a content addressed store (keyed by hash), and each venv's site packages is populated by hardlinking into that store instead of copying. Multiple venvs end up sharing the same on disk installation of a package, while using considerably less disk space overall. I also wondered if avoiding the actual file copy work could speed up install times, not just save disk space.

I've implemented it as an opt in config option, and it's best effort only: cases like editable installs fall back to a regular install rather than trying to hardlink.

Curious if this has come up before. I'm new to the community and haven't found any prior discussion of it. I have a working prototype on my fork if anyone wants to take a look:

polfeliu#1

Is this something or a similar feature that could be added to mainline Poetry? Happy to contribute and iterate on it if there's interest.