Support poetry virtual environments

Motivation

Support for poetry environments is currently our highest upvoted feature request on GitHub. Poetry attempts to solve problems with all predating tools used to manage Python packages: virtualenv, pip, pipenv, etc. by being an all-in-one tool, and hence is becoming increasingly popular. Thus adding Poetry support to the extension makes sense.

Scope

• Discover environments created using Poetry
• Identify if a provided custom interpreter path belongs to a Poetry environment
• Install dev tools using Poetry when related poetry environment is selected

Things we decided we need not address:

• Activate environment using poetry CLI: poetry shell command is similar to pipenv shell, which has its issues #4404. Besides, we don't need to activate these environments via poetry, as it manages venv/virtualenv environments which can directly be activated using the pre-existing activation scripts.
• Run scripts using poetry CLI: poetry run command can be useful to run scripts if for some reason activating poetry environment doesn't work properly. But that is not the case observed with virtual/Poetry environments for now, so we need not use it.

Design

Identifier

Global poetry environments

• Our best guess is to use the environment name pattern which Global poetry environments follow, as evident from the code here:

<sanitized_project_name>-<project_cwd_hash>-py<major>.<micro>

Implementation details behind <sanitized_project_name> and <project_cwd_hash> are too much to rely upon, so for our purposes the best we can do is verify if the pattern matches:

<anything>-<anything>-py<number>.<number>

• Should be a virtualenv/venv type environment.

If this doesn't work well, we can follow other alternatives mentioned in the spike.

Local poetry environments

• Environment folder is named .venv
• The pyproject.toml alongside the .venv folder is valid and contains a poetry section. We can either,
  • Read the pyproject.toml and see if contains the string [tool.poetry]. To be more strict, we can parse the toml file using using npm package toml and get the key tool.poetry instead.
    Note we cannot use poetry check command as that also checks parent directories for pyproject.toml files, when we only need to check the current directory.
  • Or we can run poetry env info -p with cwd set to the folder containing .venv. That should return the path to the .venv folder, which automatically validates the toml file.
• Poetry configuration allows local virtual environments. This can automatically be verified using the command. Running command for every .venv-named folder should not be much expensive, so we use poetry env info command for now.

Discovery

Our hypothesis is that users likely don’t use poetry environments created for other folders when choosing environment for the current folder (poetry itself doesn’t provide CLI to do that). It only makes sense to use poetry CLI for installation, activation etc. if the selected poetry environment was created for the workspace. Hence for now, we choose to only discovery poetry environments associated with the active projects.

In case this assumption changes in the future, we can follow this alternative to discover all poetry environments regardless of opened workspaces.

We have one workspace locator which looks into each workspace folder:

• Run poetry env list to list all environment associated with the folder.
• Unfortunately there's a poetry bug due to which poetry env list sometimes doesn't list all envs. It misses out the local activated .venv folder in some cases. So for now we also explicitly look for interpreters in .venv folder if it exists.

Use identifier, pyvenv.cfg files and other utils to get type, version and other details.

API: Each locator extends FSWatchingLocator for file watching, and uses the existing ILocator interface.

Installer

API: It uses the existing IModuleInstaller interface which exposes two methods:

• isSupported: Returns whether we can install dev tools using poetry for a particular resource. If the selected environment for the resource is the active poetry environment which was created for the resource, return true, otherwise return false. Steps,

  • Get the associated poetry environment for the resource using poetry env info -p command.
  • Verify if it's the same as the selected environment for the resource.

  Running command here is not expensive as we already have the extension activated, and this only happens after user-consent. But we can use other alternatives if for some reason this doesn't work out.

• installModule: Use poetry add --dev <moduleName> to install dev tools. Add --allow-prereleases for packages for which released version is not available yet. (Example, black)

Note that users can specify the poetry executable to use using the python.poetryPath setting.

Special Considerations and possible poetry asks

We're depending on certain internal implementation details of poetry where appropriate poetry CLI is not available:

• The way global environment directories are named: <anything>-<anything>-py<number>.<number>. Also, name of local .venv folder for identification.

  Possible resolution: Pipenv environments have a .project file, using which we can locate the project for which the current environment is created. If we have a similar feature for poetry we need not rely on the internal env naming pattern.

• Name of local .venv folder for discovery.

  Possible resolution: Wait for poetry to fix poetry-issues-3939 mentioned in the discovery section.

Telemetry and Experimentation

We need not acquire doesn't require any specific telemetry for poetry locator that we don't gather for other locators. Success of the feature can be measured along with other locators using the pythonDiscoveryModule experiment.

Testing Strategy

For discovery, unit tests uses real files to mock the file system, set the environment variables, and stubs shell execution. The corresponding ILocator APIs are then called to verify if we're getting all the environments. For installation, we have unit tests to stub shell execution, active interpreter and verify the implementation.