Python dependency management

[dhruvkb]

Should we use Pipenv or Poetry to manage Python dependencies in the project? If yes, which?

[dhruvkb]

Personally, I lean towards yes and Pipenv (but that's just because I've never used Poetry so far).

[dhruvkb]

As a proof of concept, I set the project up with Pipenv in a fork here.

[mathemancer]

I'm slightly biased against pipenv based on the fact that their emoji usage can make some terminals crash in some (rare) cases. That said, I haven't used poetry at work before, just experimented with some personal projects, but I do like that it works properly with the pyproject.toml (though there may be ways to merge the Pipfile with that easily if desired).

[dhruvkb]

If you're main complaint against Pipenv is the use of emojis, that can be turned off with an env var.

[kgodey]

Between the two, I'd lean towards Poetry, despite also never having used it before. See https://github.com/python-poetry/poetry#what-about-pipenv. I've had some issues with pipenv being really slow, so have others, see pypa/pipenv#2873 and related issues.

See also https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/.

However, I'm not convinced that we need to use either, we have requirements.txt and pip. Why use another tool at all?

[dhruvkb]

At the risk of sounding like an advocate of Pipenv, I use Pipenv regularly in all my Python projects, including Django + DRF based ones, and have not experienced any of the problems, so the cases they occur in would most likely be disjoint from mine.

The main reason I recommend a package-manager is mostly so that

• we need only list the top-level dependencies in the Pipfile
• the complete details of all frozen packages can be stored separately in Pipfile.lock

Comparing this to requirements.txt, we'll need to either manually write in the requirements file to ensure that only top-level dependencies are stored there (and risk package version differences in environments) or use pip freeze which will write all packages with exact versions (making it harder to upgrade packages).

[mathemancer]

A couple of points in favor of pipenv/poetry on top of just pip:

• hashes of dependencies for security (i.e., the downloads can be verified to be the same for a given version)
• removing packages along with the things they pulled in as dependencies is easier than with just pip

[mathemancer]

point two isn't as relevant in our use case, though, since we'll have a fresh build in the docker container (or a fresh virtualenv)

[dhruvkb]

I would not say discard point two as irrelevant to our case. If we remove a package, we might still be leaving it's dependencies behind in the requirements generated by pip freeze (unless we write the requirements.txt file by hand).

[mathemancer]

Valid point

[kgodey]

I did a bit more research and we should use pip-tools: https://github.com/jazzband/pip-tools. We'll need to maintain a requirements.in by hand, but I think that's a good thing since it's best to be very intentional about dependencies.

Here's a good article comparing the options: https://hynek.me/articles/python-app-deps-2018/

[silentninja]

I am leaning towards poetry since I have been using poetry and haven't faced any major problems so far. I like pip-tools too, would be a nice alternative if we don't need any advanced dependency management(doubtful, especially with plugins feature for mathesar)

Mathesar has moving parts like csv module, db module(would prefer for it to be a standalone library) suitable to be separate from the core and with more modular features coming in, a mono repo is much suitable for the project. Compared to other dependency managers, monorepo support is actively considered and supported by poetry.