Deprecate or remove conda develop

[jjpr-mit]

The conda develop command has been informally deprecated multiple times:

• how to use conda develop? #1992 (comment)
• conda develop . broken? python 3.5 vs 3.6 problem? #2633 (comment)

But it is still present in the code and in the documentation. That is very likely misleading to anyone trying to learn best practices for scientific computing.

Example scenario: As soon as a new scientific programmer gets comfortable enough with Python to start working on multiple projects, the next logical step is to learn how to manage environments and dependencies. They'll start by adding a setup.py with dependencies to each project, and most likely they'll start using conda to manage environments. (That's what I recommend to users of the servers I manage.) They'll create a conda environment, activate it, and then run pip install -e ., and the code from their project and all its dependencies will be available to anything they run in that environment. If they use Jupyter, they'll install that environment as a kernel.

And then, as development progresses, they'll add more stuff. They'll conda install more packages. And now they have a problem: as described in the current documentation, it's dangerous to interleave conda install and pip install. Inevitably they'll end up with an environment that's inconsistent, un-upgradeable and unusable.

On their next project, they'll try to avoid that problem. The logical solution is to use whatever is the equivalent of pip install -e in conda. Which will lead them to conda develop, which looks like the solution to their problem. They will then waste a day or more trying to get conda develop to work, eventually stumbling across #1992 or #2633. And then they will curse the name of the person who exhorted them to use conda and follow best practices. Which in several cases has been me.

Please deprecate or remove conda develop.

[astrojuanlu]

I arrived here after trying to figure out what conda develop is because it's mentioned in the SciPy developer documentation. If it's not maintained, the best thing to do is indeed to deprecate and remove it.

The logical solution is to use whatever is the equivalent of pip install -e in conda.

This was my go-to solution too, but as mentioned by @rgommers in this scipy-dev thread, one needs to be careful with at least two things:

• Build isolation. The presence of a pyproject.toml triggers build isolation, and this can interfere with the carefully created environment (for example, through conda). The solution is to opt out by adding --no-build-isolation, as explained in the docs.
• Dependency installation. pip is an installer, and its default behavior is, naturally, to install the dependencies of a project. However, pip install might meddle with the development environment by upgrading or downgrading some of the dependencies. Traditionally, setup.py develop has covered the use case of "build extensions in-place + create an editable install" without adding dependencies, but direct setup.py invocations are now deprecated. The solution, then, is to add the --no-deps flag, as explained in this somewhat hidden corner of the pip docs.

In sum, the closest replacement to python setup.py develop or conda develop . these days is probably:

pip install --no-build-isolation --no-deps -e .

(I might be missing other deviations from setup.py develop but I think I have captured the two most important ones)

[EricThomson]

I am confused. Has conda provided a canonical source of guidance on this (outside of offhand comments from individuals in issue threads)? It seems extremely important to address, once someone is making the move from user of packages to contributor.

[astrojuanlu]

I am confused. Has conda provided a canonical source of guidance on this (outside of offhand comments from individuals in issue threads)?

To my knowledge, no. That would be the first point to address.

[maximlt]

I'm watching this issue since I've been wondering about some time already about the status of conda develop. Thanks @astrojuanlu for your analysis, in the holoviz group we've also been using pip install --no-build-isolation --no-deps -e . to install a package in dev mode in a conda environment that already included all its dependencies.

Maybe @jezdez would know a little more about the status of conda develop?

[rgommers]

Imho it's not necessarily Conda's job to do this. All that's needed is a simple way to install a .pth file into site-packages pointing at the location where the development version of a package is installed if that is outside of site-packages itself. This only has to be done once, and not on every rebuild.

This could be done by pip or installer, by a build backend, or by a standalone utility. For Meson I'll have to deal with this at some point, because development installs are not going to be inplace - so conda develop . wouldn't be doing the right thing anyway (nor pip, unless the build backend told it the correct path). It's pretty low on my wish list, but plan to revisit at some point around the next SciPy release.

[Nodd]

I am using pip ìnstall -e . in a conda environment. It works quite well (even with cython builds), but the problem I have is with the dependencies. Pip checks and installs missing dependancies in the conda environments, which creates a mess. I have to install dependancies beforehand as a manual step, and then I install with pip in development mode. This features kind of makes it a conda job...

[chbrandt]

@Nodd did you check this comment: #4251 (comment) ?