Auto-detect `pynvim` located in dedicated virtual environment

[drmikehenry]

I'd like Neovim to automatically detect pynvim even when it has been installed in its own virtual environment. Finding the python interpreter executable associated with pynvim can be problematic when supporting a shared Vim configuration for multiple users on multiple platforms, as user preferences and operating system variations lead to a variety of different locations for the virtual environment.

The standard method for finding executables that have been installed in arbitrary locations is the PATH environment variable. When the executable name is unique, it may be anywhere on PATH, and the executable's actual location need not be configured.

To make it possible to detect the pynvim virtual environment's python interpreter, I've been using the below Python shim and exposing it on PATH as the executable named pynvim-python-interpreter:

import subprocess
import sys


def main() -> None:
    subprocess.run([sys.executable] + sys.argv[1:])

pynvim-python-interpreter chains to the Python interpreter associated with the virtual environment where it was installed. pynvim is a dependency that is also installed in that environment. The only Neovim configuration needed is to set g:python3_host_prog to pynvim-python-interpreter.

See https://github.com/drmikehenry/pynvim-python-interpreter for details on the implementation (published on PyPI as well). This makes installation straightforward: uv tool install pynvim-python-interpreter

If a similar shim were shipped as part of pynvim itself, then pynvim could be installed using uv tool install pynvim. I've saved the shorter name pynvim-python as a suggestion for pynvim to use in lieu of pynvim-python-interpreter.

Neovim could automatically detect this interpreter and set g:python3_host_prog via something like:

if !exists('g:python3_host_prog') && executable('pynvim-python')
    let g:python3_host_prog = 'pynvim-python'
endif

With those changes, users could uv tool install pynvim to place pynvim in a dedicated virtual environment and require no additional Neovim configuration.

[justinmk]

This repo can't really address that. This should probably be tracked in the core Nvim repo?

[drmikehenry]

The probing for pynvim-python would indeed be part of the Neovim core. pynvim seems like the natural place to expose the pynvim-python entrypoint, which is why I posted here first to gauge interest in the idea. If pynvim were to expose its Python interpreter on the PATH under the name pynvim-python and Neovim were to probe for that, it would allow the user to place pynvim in a separate virtual environment without requiring additional configuration. My separate repo gets me partway to that ideal situation; I can install pynvim-python-interpreter in different locations (depending on operating system or other constraints), and then configure g:python3_host_prog to a fixed value. But my repository is an unofficial solution that requires separate configuration, whereas if the pynvim project were to adopt this idea in some form or another and Neovim were to add support for it, then users could be told to install pynvim as a tool in their favorite manner (uv tool install pynvim, pipx install pynvim, etc.) and it would be detected automatically without further configuration. If you see problems with the idea or don't find the benefits worth the effort, that's fine; I'll be content for my own purposes with my unofficial solution.

[justinmk]

If you see problems with the idea or don't find the benefits worth the effort, that's fine; I'll be content for my own purposes with my unofficial solution.

I didn't look at your code, but the general idea would be a welcome improvement in core.

[drmikehenry]

The code itself is just the six lines of Python I'd posted; the non-boilerplate logic is a one-liner that chains to the Python interpreter via subprocess.run([sys.executable] + sys.argv[1:]). I've created pynvim-python branches for neovim and pynvim with the minimal changes needed to demonstrate the idea:

• https://github.com/drmikehenry/neovim/tree/pynvim-python
• https://github.com/drmikehenry/pynvim/tree/pynvim-python

But the branch differences are small enough to paste here:

• For Neovim: if pynvim-python is on PATH, use that name for python3_host:

  diff --git a/runtime/lua/vim/provider/python.lua b/runtime/lua/vim/provider/python.lua
  index a772b36973..62a278b2e6 100644
  --- a/runtime/lua/vim/provider/python.lua
  +++ b/runtime/lua/vim/provider/python.lua
  @@ -83,6 +83,10 @@ function M.detect_by_module(module)
       return vim.fn.exepath(vim.fn.expand(python_exe, true)), nil
     end
  
  +  if vim.fn.executable('pynvim-python') then
  +      return 'pynvim-python'
  +  end
  +
     local errors = {}
     for _, exe in ipairs(python_candidates) do
       local error = check_for_module(exe, module)

• For pynvim: publish pynvim-python as an entrypoint:

  diff --git a/pynvim/python.py b/pynvim/python.py
  new file mode 100644
  index 0000000..c408ff6
  --- /dev/null
  +++ b/pynvim/python.py
  @@ -0,0 +1,6 @@
  +import subprocess
  +import sys
  +
  +
  +def main() -> None:
  +    subprocess.run([sys.executable] + sys.argv[1:])
  diff --git a/setup.py b/setup.py
  index 55d6734..fa0f12d 100644
  --- a/setup.py
  +++ b/setup.py
  @@ -58,4 +58,9 @@ setup(name='pynvim',
         setup_requires=setup_requires,
         tests_require=tests_require,
         extras_require=extras_require,
  +      entry_points={
  +              'console_scripts': [
  +                  'pynvim-python=pynvim.python:main',
  +              ],
  +          },
         )

Using the above branches, pynvim may be installed as a Python tool via:

cd pynvim
uv tool install .

At which point pynvim-python is available on PATH:

[mike@f16:pynvim]$ which pynvim-python
/home/mike/.local/bin/pynvim-python
[mike@f16:pynvim]$ pynvim-python --version
Python 3.13.3
After installing `pynvim` via `uv tool install .`

Next, build and run neovim:

cd neovim
make CMAKE_BUILD_TYPE=RelWithDebInfo
VIMRUNTIME=runtime ./build/bin/nvim --clean

Then check the provider health in neovim via:

:checkhealth provider

This results in:

Python 3 provider (optional) ~
- `g:python3_host_prog` is not set. Searching for pynvim-python in the environment.
- Executable: /home/mike/.local/bin/pynvim-python
- Python version: 3.13.3
- pynvim version: 0.6.0dev0
- ✅ OK Latest pynvim is installed.

[drmikehenry]

Given the above proposed changes to both pynvim and neovim, do you think this is an idea that the pynvim project would accept? I recognize that there would need to be a separate request made on the neovim issue tracker and that you might not want to speak for that project without giving other maintainers a chance to weigh in. I'm happy to make proper pull requests (including documentation), but before doing that I was hoping to hear that both projects are interested. Please let me know how you'd like me to proceed.

[justinmk]

(Your if vim.fn.executable('pynvim-python') needs to check for == 1 because 0 is truthy in Lua.)

[justinmk]

Trying to understand this request. You have users that all use a "dedicated venv", but the venv isn't activated in their shell?

IIUC, your idea only works for the specific case of:

uv tool install .

right? Because uv tool does a global install from your specific venv.

So you want pynvim-python available globally, but you don't want to install your venv-specific python globally? If you did uv tool install python, wouldn't that also be a way for Nvim to automatically find the right python?

Anyway, your patch seems simple enough, so I wouldn't object to it. But we need to be able to clearly explain it, concisely.

[drmikehenry]

(Your if vim.fn.executable('pynvim-python') needs to check for == 1 because 0 is truthy in Lua.)

Thanks; I've fixed that in the branch.

[drmikehenry]

Trying to understand this request. You have users that all use a "dedicated venv", but the venv isn't activated in their shell?

That's correct. Users that want to install pynvim should use a dedicated virtual environment for that purpose. This provides desirable isolation. Activating such virtual environments is typically undesirable for reasons explained later.

IIUC, your idea only works for the specific case of:

uv tool install .

right? Because uv tool does a global install from your specific venv.

With the proposed changes, it would be uv tool install pynvim, installing pynvim from PyPI. It's not installing from a virtual environment, but installing the pynvim package to a virtual environment.

And yes, pynvim-python will end up on PATH only when uv tool install pynvim or pipx install pynvim is used.

So you want pynvim-python available globally, but you don't want to install your venv-specific python globally?

That's correct. If the python interpreter from the pynvim virtual environment were installed globally on the PATH under the name python, it would intefere with the system python interpreter.

If you did uv tool install python, wouldn't that also be a way for Nvim to automatically find the right python?

uv tool install PACKAGE_NAME is for installing Python packages. uv tool install python would try to install a Python package named python rather than expose the executable python in some way.

Neovim probes for Python interpreters by enumerating a list of standard interpreter names, looking for them along PATH. Activating an unrelated virtual environment changes the first-found interpreter, so Neovim won't find the one with pynvim anymore. But using the unique name pynvim-python for the interpreter ensures that it can be found when other virtual environments are activated.

Anyway, your patch seems simple enough, so I wouldn't object to it. But we need to be able to clearly explain it, concisely.

The below section is a short outline of the idea; the next (way too long) section contains the full details to make sure (hopefully) that everything is clear in order to properly vet the idea.

Quick description

• Neovim's :help python-provider suggests this installation method:

  python3 -m pip install --user --upgrade pynvim
  

  This fails with modern Python unless the user also provides the --break-system-packages switch. This is because installing packages "user-wide" is a deprecated practice.

• Neovim's :help python-virtualenv shows how to use a Python virtual environment dedicated to pynvim:

  If you plan to use per-project virtualenvs often, you should assign one
  virtualenv for Nvim and hard-code the interpreter path via
  |g:python3_host_prog| so that the "pynvim" package is not required
  for each virtualenv.
  
  Example using pyenv: >bash
      pyenv install 3.4.4
      pyenv virtualenv 3.4.4 py3nvim
      pyenv activate py3nvim
      python3 -m pip install pynvim
      pyenv which python  # Note the path
  The last command reports the interpreter path, add it to your init.vim: >vim
      let g:python3_host_prog = '/path/to/py3nvim/bin/python'
  

• The above requires installation of the separate tool pyenv; use of this tool is rapidly being replaced by uv, though it's certainly still usable.

• It also requires the user to hard-code a value for g:python3_host_prog, which may be non-trivial because the location may vary across operating system and individual computer.

• It's desirable for Neovim to automatically detect the location of pynvim when installed in a virtual environment.

• The standard method for using virtual environments for Python programs is to install them with pipx, uv, or similar tooling. Leveraging that standard tooling is desirable.

• With the proposed extensions to Neovim and pynvim, g:python3_host_prog need not be configured. Instead:

  • The user installs pipx or uv.

  • The user installs pynvim via one of:

    pipx install pynvim
    
    uv tool install pynvim
    

    This will expose the associated Python interpreter on PATH under the name pynvim-python.

  • Neovim will find and use pynvim-python.

  • If the user activates an unrelated virtual environment, Neovim will continue to correctly find and use pynvim-python.

Complete details

• These are described from a Linux point of view for concreteness, but the concepts apply across operating systems.

• When Python itself is installed, it creates a Python "installation" comprising:

  • A bin directory that's on the user's PATH (e.g., /usr/bin).

  • A Python interpreter in the bin directory, typically named python3 or python.

  • The Python standard library comprising a set of packages or libraries usable from the Python language.

• A package is used by a Python program by importing it, e.g. import some_package; some_package.some_function().

• The Python Package Index (PyPI, at https://pypi.org) houses packages for download and installation.

• The pip command is frequently included in a Python installation as a way to download Python packages (typically from PyPI) and install them.

• Additional packages are installed into the site area of the Python installation, such that they may be imported by the Python interpreter.

• Packages may depend on other Python packages; pip installs these in the site area as well.

• A Python package might contain a complete program, rather than just a library of code. It could contain a main() function that must be run by the Python interpreter (perhaps via import some_package; some_package.main()).

• To make program distribution convenient, a package can declare a function (e.g., main() above) as an "entry point" to be exposed as some_command for the user to run; when the package is installed (e.g., via pip install some_package), pip will synthesize a small shell script and place it in the bin directory, thus adding some_command to the PATH. Often, some_package is named the same as some_command; for example, black is a program for Python source code reformatting. After pip install black, the shell script black will be installed in the bin directory such that the user may run the command black file_to_reformat.py.

• Different packages may require conflicting dependencies; this makes it risky to install too many things into the same site packages area, as newly installed dependencies might overwrite older ones of a different version.

• For this reason, users are heavily discouraged from performing pip install as root in order to put packages into the main Python interpreter area; instead, only the system package manager should be used to install such packages, as it's the job of OS maintainers to ensure such packages are compatible.

• While users can in theory install packages into per-user site package areas in their home directories, this still leads to dependency conflicts as the number of installed packages grows. This is a big enough problem that modern Python disallows it by default. In Neovim's :help python-provider, the recommended installation method is to run python3 -m pip install --user --upgrade pynvim; on Ubuntu 24.04 with Python 3.12, this fails with the following message:

  error: externally-managed-environment
  
  × This environment is externally managed
  ╰─> To install Python packages system-wide, try apt install
      python3-xyz, where xyz is the package you are trying to
      install.
  
      If you wish to install a non-Debian-packaged Python package,
      create a virtual environment using python3 -m venv path/to/venv.
      Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
      sure you have python3-full installed.
  
      If you wish to install a non-Debian packaged Python application,
      it may be easiest to use pipx install xyz, which will manage a
      virtual environment for you. Make sure you have pipx installed.
  
      See /usr/share/doc/python3.12/README.venv for more information.
  
  note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
  hint: See PEP 668 for the detailed specification.
  

• Python provides a way to create additional site package areas to provide isolation and avoid dependency conflicts. These are called Python Virtual Environments (https://docs.python.org/3.13/library/venv.html), as mentioned in the error message above.

• A virtual environment lives in a separate file tree. It shares the same Python interpreter and standard library packages as the "base" Python upon which it is built. python -m venv path/to/some_venv creates a virtual environment at an arbitrary location; it will be based on the Python installation of the python interpreter. some_venv/bin will hold the Python interpreter and (typically) the pip command; but this bin directory is not automatically added to the user's PATH.

• some_venv/bin/pip install some_package will install some_package into the site packages area of some_venv, rather than into the Python installation area of the base Python.

• If some_package contains an entry point for some_command, then some_venv/bin/pip install some_package creates the shell script some_venv/bin/some_command. This shell script has a shebang with the absolute path to the Python interpreter (e.g., #!/abs/path/to/some_venv/bin/python), such that the script will run with the correct interpreter regardless of the value of PATH. In other words, running /abs/path/to/some_venv/bin/some_command will work correctly.

• A virtual environment may be "activated"; activation adds some_venv/bin to the PATH in the currently running shell and sets the VIRTUAL_ENV environment variable to the absolute path to some_venv. At most one virtual environment should be activated at a time.

• If the virtual environment is activated, then the user can just type some_command in order to such a command installed via an entry point in the virtual environment.

• Instead of activation, it's also possible to simply copy the shell script to a directory already on the PATH; the absolute path in the shebang line ensures that the script runs with the correct Python interpreter and its associated packages from the virtual environment. This adds only that shell script to the PATH; this is unlike activating the virtual environment, which exposes the entire contents of its bin directory and overrides the default Python interpreter and its packages.

• When installing a program written in Python, it's good practice to create a dedicated virtual environment for that program, install just that program into the virtual environment, then expose just the command's shell script to the PATH. This keeps the program and its dependencies isolated from other such programs, from the main Python installation, and from other virtual environments used by Python programmers as they develop new programs.

• Tools have been written to automate installation of such Python programs. One such tool is pipx; the pipx documentation (https://pipx.pypa.io/latest/how-pipx-works/) explains the steps it takes. Another such tool (which is rapidly gaining in popularity) is uv (https://docs.astral.sh/uv/guides/tools/). After installing either pipx or uv, the user can install a Python-based program such as black in one step (pipx install black or uv tool install black), after which the new command black is available on PATH for immediate use.

• When developing a Python program, it's good practice to create a virtual environment for the development. Only those packages necessary for running and developing the program should be installed into the virtual environment. A pyproject.toml file declares the full list of necessary packages, such that development tools (like pip and uv) know what packages to install.

• For Python support, Neovim requires a Python interpreter that can import pynvim. If pynvim is installed via the OS package manager into the system's Python installation, then the default Python interpreter may be used. Neovim currently probes for such a Python interpreter by checking along PATH for python3, python3.13, python3.12, etc.; for each such candidate interpreter, it tries to import pynvim to find a suitable interpreter.

• The OS-supplied pynvim may be out of date, or the user may not have sufficient permission to install OS packages. The user may install pynvim into a custom virtual environment, but Neovim's probing algorithm won't find it by default.

• The user could activate the virtual environment with pynvim, but it can be disruptive to override the default Python interpreter and packages globally.

• A user developing a Python program needs a "dev" virtual environment for the program under development; activating a pynvim environment conflicts with this need. Installing pynvim into the programmer's dev virtual environment is undesirable because it changes the dev environment, and such installation must be repeated for every Python-based project under development. In addition, tools like uv may "synchronize" the environment with the expected packages from pyproject.toml, causing pynvim to become uninstalled during the course of development.

• If the default probing algorithm is insufficient, the user may set the Neovim variable g:python3_host_prog to point to the correct Python interpreter. Since the pynvim virtual environment lives at an arbitrary location, it's difficult for Neovim to probe for it.

• Neovim :help python-virtualenv suggests using a dedicated virtual environment for pynvim and hard-coding g:python3_host_prog to the associated Python interpreter:

  Example using pyenv: >bash
      pyenv install 3.4.4
      pyenv virtualenv 3.4.4 py3nvim
      pyenv activate py3nvim
      python3 -m pip install pynvim
      pyenv which python  # Note the path
  The last command reports the interpreter path, add it to your init.vim: >vim
      let g:python3_host_prog = '/path/to/py3nvim/bin/python'
  

  It also points to https://github.com/zchee/deoplete-jedi/wiki/Setting-up-Python-for-Neovim which makes a similar recommendation.

• Though the user can set g:python3_host_prog as part of the Neovim configuration, this is an extra step. The location of the pynvim virtual environment may also vary by operating system and by particular computer, complicating the configuration. In addition, for Neovim configurations shared by multiple users, per-user preferences may causes additional variation.

• Combining the above considerations yields a solution that allows Neovim to probe for the location of pynvim while keeping pynvim in an isolated virtual environment:

  • In addition to the names python3, python3.13, etc., Neovim will probe for the interpreter under the name pynvim-python.

  • pynvim will use an entry point to expose the command pynvim-python. This is a small Python program that just chains to the virtual environment's Python interpreter.

  • If pynvim is installed using pipx install pynvim or uv tool install pynvim, then those tools will expose pynvim-python on the PATH where Neovim can probe for it.

[justinmk]

I read everything except the "Complete details" :D And it sounds like a great plan. Yes, the pyenv docs should be updated to mention uv and/or pipx, please feel free to update them.

This will hopefully fix a lot python provider and/or checkhealth issues:
https://github.com/neovim/neovim/issues?q=is%3Aissue%20state%3Aopen%20python%20checkhealth