Python & Jupyter extension fail to detect upgraded Python

[paul-rogers]

Type: Bug

Behaviour

After an OS upgrade replaced an old Python with a new version, VS Code fails to detect the new version.

Steps to reproduce

I use Linux Mint. Version 21 included Python 3.10.12 with symlinks in /bin/python3 and /usr/bin/python3. VS code detected these and things worked just fine.

I recently upgraded to Mint version 22, which removed the Python 3.10.12 and replace it with Python 3.12.3. In a shell:

> which python3
/usr/bin/python3
> python3 --version
Python 3.12.3

However, if I use Python: Select Interpreter I get the following:

Notice that VS Code still believes that the system Python is 3.10.12. Just to be clear: I have restarted VS Code multiple times since the OS upgrade.

I reviewed the documentation. This document says that Python will detect the Python versions automatically. Clearly, it did not.

Python 3.10.12 no longer exists on the system. Yet, the Python extension seems to believe that the Python in /bin/python3 is that version. Perhaps the version information is cached. However, the documentation provides no hints (which I could find) to tell VS Code to flush its cache and to go out and look to see which version is actually installed.

At the same time, Jupyter fails to detect this version also. Create a new notebook. Select "Select Interpreter." Only the non-existent 3.10.12 interpreter is listed. Now, it could be that Jupyter fails to find the system Python because that version is managed by the OS and I've not installed the required Python modules, though the documentation says that I'll be prompted to install them.

My workaround was to create a virtual environment (in a terminal). In a terminal, with the virtual environment added to the path:

> which python
/home/paul/pyenv/3.12/bin/python
> python --version
Python 3.12.3

The documentation says that this virtual environment will be detected automatically, since it is in ~/pyenv. However, it was not. I was able to configure it via Python: Select Interpreter/Enter Interpreter Path.

Then, I had to install IPyKernel by hand after which I could finally select the correct interpreter in a new Juypter notebook.

Specific Issues

1. Neither the Python nor Jupyter extensions have noticed that Python was upgraded.
2. It may be that the displayed version number is just cosmetic and does not affect operation. Still, it would be good for it to be correct.
3. The Python extension did not find my virtual environment as promised: I had to configure it manually.
4. The Jupyter extension did not find my virtual environment and install the IPyKernel module as promised in the documentation.
5. Either provide information on how to diagnose such problems (perhaps turn on logging somewhere), or if there is no current mechanism, please provide one that tells me exactly where the Python and Jupyter extensions are searching, why, and why they choose to not find my interpreters.

Diagnostic data

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

Extension version: 2024.16.1
VS Code version: Code 1.95.0 (912bb683695358a54ae0c670461738984cbb5b95, 2024-10-28T20:16:24.561Z)
OS version: Linux x64 6.8.0-47-generic
Modes:

• Python version (& distribution if applicable, e.g. Anaconda): 3.12.3
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
• Value of the python.languageServer setting: Pylance

User Settings

languageServer: "Pylance"

testing
• unittestArgs: "<placeholder>"
• unittestEnabled: true

Installed Extensions

Extension Name	Extension Id	Version
Code Spell Checker	streetsidesoftware.code-spell-checker	3.0.1
Debugger for Java	vscjava.vscode-java-debug	0.58.0
Extension Pack for Java	vscjava.vscode-java-pack	0.29.0
GitLens — Git supercharged	eamodio.gitlens	15.6.2
Gradle for Java	vscjava.vscode-gradle	3.16.4
IntelliCode	VisualStudioExptTeam.vscodeintellicode	1.3.2
IntelliCode API Usage Examples	VisualStudioExptTeam.intellicode-api-usage-examples	0.2.9
JavaScript Debugger	ms-vscode.js-debug	1.95.1
JavaScript Debugger Companion Extension	ms-vscode.js-debug-companion	1.1.3
Jupyter	ms-toolsai.jupyter	2024.10.0
Jupyter Cell Tags	ms-toolsai.vscode-jupyter-cell-tags	0.1.9
Jupyter Keymap	ms-toolsai.jupyter-keymap	1.1.2
Jupyter Notebook Renderers	ms-toolsai.jupyter-renderers	1.0.20
Language Support for Java(TM) by Red Hat	redhat.java	1.35.1
markdownlint	DavidAnson.vscode-markdownlint	0.56.0
Maven for Java	vscjava.vscode-maven	0.44.0
Project Manager for Java	vscjava.vscode-java-dependency	0.24.0
Pylance	ms-python.vscode-pylance	2024.10.1
Python	ms-python.python	2024.16.1
Python Debugger	ms-python.debugpy	2024.12.0
Rainbow CSV	mechatroner.rainbow-csv	3.12.0
Rewrap	stkb.rewrap	1.16.3
Scientific Terms - Code Spell Checker	streetsidesoftware.code-spell-checker-scientific-terms	0.2.2
Table Visualizer for JavaScript Profiles	ms-vscode.vscode-js-profile-table	1.0.10
Test Runner for Java	vscjava.vscode-java-test	0.42.0

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz (8 x 4429)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: disabled_software vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off webnn: disabled_off
Load (avg)	3, 2, 2
Memory (System)	31.31GB (10.72GB free)
Process Argv	--crash-reporter-id 5e232e7b-355a-468c-ac4a-02c300f898bd
Screen Reader	no
VM	0%
DESKTOP_SESSION	cinnamon
XDG_CURRENT_DESKTOP	X-Cinnamon
XDG_SESSION_DESKTOP	cinnamon
XDG_SESSION_TYPE	x11

A/B Experiments

vsliv368cf:30146710
vspor879:30202332
vspor708:30202333
vspor363:30204092
vscod805:30301674
binariesv615:30325510
vsaa593:30376534
py29gd2263:31024239
c4g48928:30535728
azure-dev_surveyone:30548225
a9j8j154:30646983
962ge761:30959799
pythongtdpath:30769146
pythonnoceb:30805159
asynctok:30898717
pythonmypyd1:30879173
h48ei257:31000450
pythontbext0:30879054
cppperfnew:31000557
dsvsc020:30976470
pythonait:31006305
dsvsc021:30996838
g316j359:31013175
dvdeprecation:31068756
dwnewjupytercf:31046870
2f103344:31071589
impr_priority:31102340
nativerepl1:31139838
refactort:31108082
pythonrstrctxt:31112756
wkspc-onlycs-t:31132770
nativeloc1:31134641
wkspc-ranged-t:31151552
cf971741:31144450
iacca1:31156133
notype1cf:31157160
5fd0e150:31155592
dwcopilot:31170013

[karthiknadig]

We typically don't re-run detection is some of these cases. A workaround is to run Python: Clear Cache and Reload command.

[paul-rogers]

@karthiknadig, thanks for the explanation. I ran that command and it did, indeed "forget about" the old Python versions and found the updated version. Unfortunately, if also forgot about my virtual environment and did not find it when I selected Python: Choose Interpreter. I had to configure that by hand with Enter interpreter path. Recall that I placed my virtual environment in a path location that the documentation says that VS Code will search: ~pyenv/3.12. Any reason that VS Code would not find that virtual environment?

It would be helpful to add to the documentation the steps needed when the system Python is upgraded:

1. Update the system Python.
2. Python: Clear Cache and Reload
3. Python: Select Interpreter and create/configure a new virtual environment (if needed)
4. In each notebook, Select Interpreter to pick one of the newly available interpreters.

Step 3 above is needed in systems like Linux Mint (Ubuntu base) where the system interpreter is "managed" and does not easily allow installing user-specific packages.

[karthiknadig]

You should be able to see in the logs for Output > Python and Output > Python Locator to see what it is looking for.

[paul-rogers]

@karthiknadig, thanks for the suggestion. It seems that the internals found the virtual environment, but not the Python: Select Interpreter dialog nor the Jupyter Select Kernel dialog. Any idea why this might be?

I ran another Python: Clear Cache and Reload. Here are the relevant log lines:

2024-10-30 12:07:11.021 [info] Starting Python Locator /home/paul/.vscode/extensions/ms-python.python-2024.16.1-linux-x64/python-env-tools/bin/pet server
2024-10-30 12:07:11.536 [info] Discovered manager: (Conda) /home/paul/anaconda3/bin/conda
2024-10-30 12:07:11.537 [info] Discovered env: /home/paul/anaconda3/bin/python
2024-10-30 12:07:11.577 [info] Discovered env: /bin/python3
2024-10-30 12:07:11.636 [info] Discovered env: /usr/bin/python3
...
2024-10-30 12:07:11.665 [info] Locator PyEnv took 2.126647ms
...
2024-10-30 12:07:11.705 [info] Resolved Python Environment /home/paul/pyenv/3.12/bin/python
2024-10-30 12:09:15.754 [info] Resolved Python Environment /home/paul/pyenv/3.12/bin/python
2024-10-30 12:09:15.794 [info] Discovered manager: (Conda) /home/paul/anaconda3/bin/conda
2024-10-30 12:09:15.794 [info] Discovered env: /home/paul/anaconda3/bin/python
2024-10-30 12:09:15.797 [info] Discovered env: /usr/bin/python3
2024-10-30 12:09:15.801 [info] Discovered env: /bin/python3
...
2024-10-30 12:09:15.853 [info] Locator PyEnv took 363.697µs
...
2024-10-30 12:13:07.849 [info] Resolved Python Environment /home/paul/pyenv/3.12/bin/python
2024-10-30 12:19:29.004 [info] Resolved Python Environment /home/paul/pyenv/3.12/bin/python
2024-10-30 12:19:29.044 [info] Discovered manager: (Conda) /home/paul/anaconda3/bin/conda
2024-10-30 12:19:29.045 [info] Discovered env: /home/paul/anaconda3/bin/python
...
2024-10-30 12:37:33.179 [info] Resolved Python Environment /home/paul/pyenv/3.12/bin/python
(Above repeated about 40 times)

The whole process seems to have been repeated a couple more times though I triggered a refresh only once. The above suggests that the code did find my environment. In fact, the code did so a total of about 50 times. (Why so many?) However, if I then do Python: Select Interpreter, this is what I see:

Notice that the virtual environment does not appear on the list though it is marked as the selected interpreter.

The expected behavior is that, if the code did discover my virtual environment, that it would be available as one of the interpreters so I can select it. Any reason that it does not appear in the UI interpreter list?

A subtle point is that the first time I did the above, my virtual environment was not selected. The second time, the virtual environment selection appeared to "survive" the Python: Clear Cache and Reload, so that, even though it does not appear in the list, it is still marked as the current interpreter.

All of this would just be an annoying UI glitch if it were not then for Jupyter. Using Python: Clear Cache and Reload causes a Jupyter
notebook to "forget about" my virtual environment, which was the selected kernel. As noted above, the Python: Select Interpreter says my virtual environment is the selected one. But Jupyter knows nothing about it. Click Select Kernel and this is what I see:

Now, to be complete, let's select another interpreter using Python: Select Interpreter: the recommended one. Again run Python: Select Interpreter. The system interpreter is shown selected, but the rest of the list looks identical to the above screenshot: the selected interpreter is still in the list and my virtual environment does not appear.

Let's try one more trick. Again run Python: Select Interpreter and select \usr\bin\python3. (This is just a symlink to the other one, probably for backwards compatibility with older Linux setups.) Now, yet again click Select Kernel in the Jupyter notebook. This selected kernel does not appear, I see the same list as in the image above. Why would the selected interpreter fail to appear? Since it is just a symlink, it is identical to the interpreter which does appear. Moreover, why does the Jupyter list not contain the Anaconda entry in the Python: Select Interpreter list?

Finally, I used Python: Select Interpreter/Enter interpreter path and manually selected my environment. That environment still does not appear in the Python: Select Interpreter list. Why? This means that if I were to switch to another interpreter for some reason, VS Code will again "forget" about my virtual environment.

Fortunately, after I manually configure the virtual environment, it does then show up in the above Jupyter Select Kernel list. Why does this environment show only in Juypter, but not in Python?

In fact, once I configure my environment manually, the notebook then automatically selects that kernel, likely because the kernel name is stored in the notebook itself. Thus, the Python: Clear Cache and Reload command doesn't appear to clear the kernel associated with notebooks, it just makes those kernels unavailable to Jupyter if the kernel configured in Jupter is not in the refreshed list.

Bottom line: this is all rather fiddly and could, perhaps, be streamlined a bit to avoid the need to ask for help for such a simple task. To be clear, the following would be helpful:

1. Fix the bug that prevents the discovered virtual environment from showing in the list of interpreters.
2. Fix the bug that prevents Jupyter from displaying the full set of interpreters (kernels) in the Python interpreter list. (If the interpreter is not usable, just show it a "misconfigured" or some such to help us users figure things out.)
3. Document the process as suggested earlier.

[karthiknadig]

@paul-rogers Can you check what python.locator is set to? try setting it to native, and see if that shows up in the list of interpreters. That might also improve the Jupyter case as it uses the same set just bit more filtered.

the locator has two options js and native. The js is the legacy locator, and native is a new locator we are experimenting with. the native locator should handle things better than js, but it could have gaps in it. With native it should also handle the updates, and changes to python installs better.

[paul-rogers]

@karthiknadig, the python.locator was already set to native. Thus, the behavior described above is associated with the newer native locator.

[karthiknadig]

@DonJayamanne Any idea what might be going on here? I am not fully familiar with how the kernel picker finds/filters things.

[paul-rogers]

FWIW, while debugging Jupyter Issue #16185 , I enabled trace logging for Jupyter. I see the following in the resulting logs:

Visual Studio Code (1.95.0, undefined, desktop)
Jupyter Extension Version: 2024.10.0.
Python Extension Version: 2024.16.1.
Pylance Extension Version: 2024.10.1.
Platform: linux (x64).
Temp Storage folder ~/.config/Code/User/globalStorage/ms-toolsai.jupyter/version-2024.10.0
Workspace folder ~/omitted, Home = /home/paul
21:45:41.154 [debug] Start refreshing Kernel Picker (1730436341154)
21:45:41.751 [trace] No controller, hence notebook communications cannot be initialized for editor ~/omitted.ipynb
21:45:41.752 [trace] No controller, hence notebook communications cannot be initialized for editor ~/omitted.ipynb
21:45:41.752 [debug] Start refreshing Interpreter Kernel Picker
21:45:43.395 [trace] Kernel Spec for 'Python 3 (ipykernel)' (~/.local/share/jupyter/kernels/python3/kernel.json) is hidden. (isDefaultKernelSpec = true, language = python, registrationInfo = undefined)
21:45:43.396 [debug] Get Custom Env Variables, Class name = Em, completed in 2243ms, has a truthy return value, Arg 1: undefined, Arg 2: "RunPythonCode"
21:45:43.396 [debug] Jupyter Paths /kernels: 
21:45:43.397 [debug] Kernel Spec Root Paths, /usr/share/jupyter/kernels, /usr/local/share/jupyter/kernels, ~/.local/share/jupyter/kernels
21:45:45.116 [trace] Python API env change detected, add => '/bin/python3'
21:45:45.117 [trace] Search for KernelSpecs in Interpreter /bin/python3
21:45:45.171 [trace] Python API env change detected, add => '/usr/bin/python3'
21:45:45.172 [trace] Search for KernelSpecs in Interpreter /usr/bin/python3
21:45:45.211 [trace] Python API env change detected, add => '/home/~/anaconda3/bin/python'
21:45:45.211 [trace] Search for KernelSpecs in Interpreter ~/anaconda3/bin/python
21:45:45.248 [trace] Hiding default KernelSpec ~/anaconda3/bin/python for interpreter ~/anaconda3/bin/python (KernelSpec file ~/anaconda3/share/jupyter/kernels/python3/kernel.json)
21:45:45.252 [debug] Refreshed Environments
21:45:45.257 [trace] Python API env change detected, add => '/home/~/pyenv/3.12/bin/python'
21:45:45.257 [trace] Search for KernelSpecs in Interpreter ~/pyenv/3.12/bin/python
21:45:45.364 [trace] Hiding default KernelSpec /python for interpreter ~/pyenv/3.12/bin/python (KernelSpec file ~/pyenv/3.12/share/jupyter/kernels/python3/kernel.json)
21:45:45.381 [debug] KernelProvider switched kernel to id = .jvsc74a57bd049d54976ddc812de6cd3e61a814bfa938ec6d96e5bed8701f21d85baa25c5f8d./home/~/pyenv/3.12/python./home/~/pyenv/3.12/python.-m#ipykernel_launcher

At which point the kernel starts the kernel for my notebook, etc.

To my naive eyes, it seems that the search did not find my virtual environment. It looks like it then does Search for KernelSpecs in Interpreter ~/pyenv/3.12/bin/python, which is the virtual environment configured in Python and used for this notebook. Then, it says that it is Search for KernelSpecs in Interpreter ~/pyenv/3.12/bin/python.

This raises two questions: why does it not locate the virtual environment during the various searches? Why does it want to "hide" the virtual environment once it finds it?

[DonJayamanne]

@paul-rogers Lets try to focus on one thing at a time.
Looks like you are running into a few issues and we're trying to solve all here.

1. My workaround was to create a virtual environment (in a terminal). In a terminal, with the virtual environment added to the path:

What is this virtual environment?
Is this what you are referring to as the virtual env ~/pyenv/3.12/bin/python
@karthiknadig this is returned by pet but not listed in the python quick pick.

2. You do not seem to have the latest version of the Python extension.
   Please can you install the latest version, reload and try again.

3. To my naive eyes, it seems that the search did not find my virtual environment. It looks
   Yes it did.
   This is further evidence of that

KernelProvider switched kernel to id = .jvsc74a57bd049d54976ddc812de6cd3e61a814bfa938ec6d96e5bed8701f21d85baa25c5f8d./home//pyenv/3.12/python./home//pyenv/3.12/python.-m#ipykernel_launcher

Why does it want to "hide" the virtual environment once it finds it?
Hiding default KernelSpec /python for interpreter ~/pyenv/3.12/bin/python (KernelSpec file ~/pyenv/3.12/share/jupyter/kernels/python3/kernel.json)

We're not hiding Envirolnments, we're hiding a kernel spec, (the file hiddent is python3/kernel.json) thats because it does't contain anything specia. and it points to the same environment. This is not an issue.

For now lets ignore Jupyter, and deal with Python extension and not seeing the environments you expect to see
Once you have resolved this, we can then focus on why things do not appear in Jupyter kernel picker.