VSCode leaves several running processes after exit

[dimitry-ishenko]

Issue Type: Bug

Steps to reproduce:

1. Launch vscode.
2. Open or create a Jupyter Notebook.
3. Do some work on it.
4. Close vscode.
5. Optional: Rinse and repeat.

Expected behavior:

Any processes started by vscode are closed.

Actual behavior:

After launching and closing vscode a few times (and realizing I am starting to run out of memory) I notice this:

user@laptop:~$ ps aux | grep -v networkd-dispatcher | grep -v unattended | grep -v grep | grep python
user   344513  0.2  0.6 4115828 102040 ?      Sl   May19   3:19 /home/user/.vscode-oss/extensions/ms-python.python-2020.5.80290/languageServer.0.5.50/Microsoft.Python.LanguageServer
user   345452  0.4  5.3 5162548 858400 ?      Sl   May19   5:46 /home/user/.vscode-oss/extensions/ms-python.python-2020.5.80290/languageServer.0.5.50/Microsoft.Python.LanguageServer
user   345607  0.2  0.6 4443632 102484 ?      Sl   May19   3:01 /home/user/.vscode-oss/extensions/ms-python.python-2020.5.80290/languageServer.0.5.50/Microsoft.Python.LanguageServer
user   353204  0.2  0.6 3788220 105468 ?      Sl   May19   1:55 /home/user/.vscode-oss/extensions/ms-python.python-2020.5.80290/languageServer.0.5.50/Microsoft.Python.LanguageServer
user   353605  0.0  0.2 117072 46508 ?        S    May19   0:01 /usr/bin/python3 /home/user/.vscode-oss/extensions/ms-python.python-2020.5.80290/pythonFiles/pyvsc-run-isolated.py vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.jupyter_daemon -v
user   353771  0.3  0.6 3796468 105576 ?      Sl   May19   2:29 /home/user/.vscode-oss/extensions/ms-python.python-2020.5.80290/languageServer.0.5.50/Microsoft.Python.LanguageServer
user   353933  0.2  0.2 3722708 39000 ?       Sl   May19   1:31 /home/user/.vscode-oss/extensions/ms-python.python-2020.5.80290/languageServer.0.5.50/Microsoft.Python.LanguageServer
user   354840  0.2  0.2 3796424 35728 ?       Sl   00:35   1:34 /home/user/.vscode-oss/extensions/ms-python.python-2020.5.80290/languageServer.0.5.50/Microsoft.Python.LanguageServer
user   355331  0.9  5.4 6179224 869260 ?      Sl   00:39   6:06 /home/user/.vscode-oss/extensions/ms-python.python-2020.5.80290/languageServer.0.5.50/Microsoft.Python.LanguageServer
user   356104  0.3  0.6 3862044 111216 ?      Sl   00:52   1:52 /home/user/.vscode-oss/extensions/ms-python.python-2020.5.80290/languageServer.0.5.50/Microsoft.Python.LanguageServer
user   366116  1.6  0.4 3796252 76032 ?       Sl   10:27   0:35 /home/user/.vscode-oss/extensions/ms-python.python-2020.5.80290/languageServer.0.5.50/Microsoft.Python.LanguageServer
user   366803  2.3  0.4 1007044 69596 ?       Sl   10:32   0:44 /usr/bin/python3 /home/user/.vscode-oss/extensions/ms-python.python-2020.5.80290/pythonFiles/pyvsc-run-isolated.py vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.jupyter_daemon -v
user   366808 10.1  5.4 5509256 871684 ?      Sl   10:32   3:16 /home/user/.vscode-oss/extensions/ms-python.python-2020.5.80290/languageServer.0.5.50/Microsoft.Python.LanguageServer
user   366831  0.0  0.2 551048 35328 ?        Ssl  10:32   0:01 /usr/bin/python3 -m ipykernel_launcher -f /home/user/.local/share/jupyter/runtime/kernel-33b1ca6c-b09c-41dd-b2fc-c466b2511581.json
user   366855  0.0  0.2 550640 35408 ?        Ssl  10:32   0:01 /usr/bin/python3 -m ipykernel_launcher -f /home/user/.local/share/jupyter/runtime/kernel-a5a7d55e-0ebc-4b4b-aac4-4d1dddff8450.json
user   368190  7.6  1.2 2471816 200372 ?      Ssl  10:45   1:27 /usr/bin/python3 -m ipykernel_launcher -f /home/user/.local/share/jupyter/runtime/kernel-b00685be-189e-4f07-80e0-237ab6772827.json
user   368212  0.0  0.2 550648 43728 ?        Ssl  10:45   0:00 /usr/bin/python3 -m ipykernel_launcher -f /home/user/.local/share/jupyter/runtime/kernel-918a11e0-f81e-4740-a3c1-e5d5ca074570.json
user   369055  4.4  1.3 2076996 219232 pts/0  Sl+  10:57   0:19 /usr/bin/python3 /home/user/.local/bin/ipython
user   369168  2.0  0.3 709300 62112 ?        Sl   10:58   0:08 /usr/bin/python3 /home/user/.vscode-oss/extensions/ms-python.python-2020.5.80290/pythonFiles/pyvsc-run-isolated.py vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.jupyter_daemon -v
user@laptop:~$ 

Extension version: 2020.5.80290
VS Code version: Code - OSS 1.46.0 (3984852b33bd6897dd83108ac305c567d49e8fda, 2020-05-15T14:46:12.527Z)
OS version: Linux x64 5.4.0-29-generic

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz (8 x 1137)
GPU Status	2d_canvas: enabled flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on oop_rasterization: disabled_off protected_video_decode: unavailable_off rasterization: disabled_software skia_renderer: disabled_off_ok video_decode: unavailable_off viz_display_compositor: enabled_on viz_hit_test_surface_layer: disabled_off_ok webgl: enabled webgl2: enabled
Load (avg)	1, 2, 2
Memory (System)	15.23GB (0.93GB free)
Process Argv	--no-sandbox --unity-launch
Screen Reader	no
VM	0%
DESKTOP_SESSION	ubuntu
XDG_CURRENT_DESKTOP	Unity
XDG_SESSION_DESKTOP	ubuntu
XDG_SESSION_TYPE	x11

[rchiodo]

Thanks. This may be unavoidable depending upon the machine. VS code gives all extensions 5 seconds to finish shutdown. If we don't get called to close our processes before that 5 seconds is up, they'll stay running.

We'll investigate though. We might not be closing the processes in the first place.

[rchiodo]

Maybe the VS code OSS version behaves differently, but I'm not reproing this on a windows or a linux box.

[dimitry-ishenko]

@rchiodo I had a similar problem even when I ran the official version. It left several ipykernel_launcher instances. The LanguageServer is a new addition to the family. I switched to it around the same time I've switched to the oss version.

I can try running the official version with LanguageServer enabled to see if I have the same problem.

[rchiodo]

Might also be other extensions installed are taking the 5 seconds that we'd normally use for closing processes. Can you try this after disabling all other extensions other than the python extension?

[dimitry-ishenko]

Disabling other extensions doesn't make a difference (vscode-oss @d5eac26). Will try with the official release next.

[dimitry-ishenko]

Well, this is pretty rad... So, I've tried the official version of vscode and it was correctly cleaning up all the processes as you said.

Next, I went ahead and completely nuked all the directories associated with vscode-oss that I could find. IIRC they were: ~/.vscode-oss ~/.cache/Microsoft and ~/.config/Code - OSS.

Then, launched vscode-oss, reinstalled the Python extension and now it seems to work fine... nothing is left behind... 😕

Now, the interesting part is that the ~/.cache/Microsoft dir is no longer being created. Maybe some bad juju inside it was causing this?...

@rchiodo if you don't mind, I'd like to keep this issue open for a few more days while I do some more intensive testing. I will come back here and close if all is good.

[bchoivw]

Also seeing ipykernel and language server keep running on remote ssh after I exit out of VSCode....not sure if it's intended or not for remote server, but for now I'm rebooting remote machine to free up RAM. Doing "remote ssh - kill vscode server on host" seem to close some processes but not jupyter nor python language server. Tried manually entering "jupyter notebook stop" but that results in timeout error. Right now my option of killing orphan process is just rebooting the machine.

[dimitry-ishenko]

In my case I am not seeing the languageServer anymore, but after heavy use of Jupyter for several hours (executing and re-executing multiple cells, restarting the kernel several times, clearing all cell output, etc.) I did notice 2 ipykernel processes left behind (one in a zombie state).

[EricChen1248]

Same issue here, two users SSHing into a GCP ubuntu server. Closing VSCode will leave many suspended processes for python language server and the ipykernel

Both users are running the official version vscode downloaded from code.visualstudio.com

[pratheekrebala]

I am also seeing the same issue on the vscode installed through apt from packages.microsoft.com. Just opening a jupyter notebook and exiting vscode leaves behind orphan processes:

.vscode/extensions/ms-python.python-2020.6.91350/pythonFiles/pyvsc-run-isolated.py vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.kernel_launcher_daemon -v

Usually 2-3 of these. FWIW: MyJupyter Server URI is set to local

[dimitry-ishenko]

This seems to have gotten worse with later versions of code-insiders as well. I will post some specifics shortly

[greazer]

Old bug with no new comments for nearly a year. Closing.

[dimitry-ishenko]

@greazer this was never fixed. I've observed it few weeks ago. Please reopen.

[dimitry-ishenko]

Thank you @rchiodo. I didn't want to keep bugging ya'll as I know everybody is busy. I am still patiently waiting for this to get fixed.

[JPvRiel]

I can confirm this behavior of leaving a stale python instance of -m ipykernel_launcher still happens with current versions of vscode on an up to date Ubuntu 20.04 OS:

code --version
1.59.0
379476f0e13988d90fab105c5c19e7abc8b1dea8
x64

And

$ code --list-extensions --show-versions | grep jupyter
ms-toolsai.jupyter@2021.8.1236758218

By the way code -s is useful to see the ipykernel_launcher instances started. Close vscode, and the python ipykernel_launcher PIDs get orphaned and picked up/owned by the systemd user session pid because vscode failed to reap those processes when closing.

To find details of stuff left behind/not cleaned up by vscode:

ps -f -p $(pgrep -f ipykernel_launcher) | more

To see exactly which IPython orphans have ended up owned by the user init instance of systemd:

ps -f -p $(pgrep -P $(pgrep -u $UID systemd) -f ipykernel_launcher) | more

Note pgrep only matches immediate child with -P; no recursion down to deeper child process of running vscode instances.

To kill/cleanup just the orphaned IPython instances (and reclaim your RAM!):

pkill -P $(pgrep -u $UID systemd) -f ipykernel_launcher

[rchiodo]

The root cause of this bug is the daemon's we spin up to make kernel's launch faster.

I believe you can workaround the problem by eliminating these files here (copy them elsewhere first in case that breaks something else):

<extension dir>/pythonFiles/lib/python/pyls_jsonrpc

[JPvRiel]

root cause of this bug is the daemon's we spin up to make kernel's launch faster.

Thanks @rchiodo

So far, removing pyls_jsonrpc seems to work:

mkdir /tmp/ipy_buggy_launch
mv ~/.vscode/extensions/ms-toolsai.jupyter-2021.8.1236758218/pythonFiles/lib/python/pyls_jsonrpc /tmp/ipy_buggy_launch

Since doing this, haven't yet reproduced orphaned IPython instances after vscode closes.

[dimitry-ishenko]

@rchiodo @JPvRiel is there a measurable downside to doing this?

[rchiodo]

Downside is longer startup times for kernels. But that's relative. On some windows machines this can cost you 3-5 seconds for startup. On a linux box, it's a lot less time.

[DonJayamanne]

I'd suggest disabling this as a first step (via setting or code).

[rchiodo]

You think there might be cases where it's still necessary?

[IanMatthewHuff]

Marking this as verified though I don't have an exact repro to test here. Just working with insiders and making sure things are cleaned up after shutdown. And I'm not seeing anything left over.

[rchiodo]

Repro case is using linux/wsl, start a notebook kernel, close VS code itself.

It used to leave the kernel processes running.

[dimitry-ishenko]

How does one know if this has landed in the insiders build? I've still experienced this today, but my build was about a week old

[rchiodo]

You need the latest jupyter prerelease extension. As of me writing this comment it would be:

[IanMatthewHuff]

Whoops my mistake. I'm on Mac, didn't notice the WSL / linux specificity. I'll remove my verification.

[dimitry-ishenko]

@rchiodo I can confirm that this handles the problem. Sorry, dunno how to add verified label... 🤷🏻‍♂️

[dimitry-ishenko]

@rchiodo any idea when this will land in the release build? Or is it safe to stay on pre-release? I am using Jupyter for work and if pre-release is unstable, I'd rather wait

[rchiodo]

Next week sometime it will be in stable. Pre-release is certainly less stable. It ships every night after running our tests.

[Winand]

Does result still depend on pyls_jsonrpc folder?
I use Jupyter extension 2022.3.1000762011 with code-server 4.3.0 (standalone) in CentOS 7.8 virtual machine.

I've tried to run code-server with and without pyls_jsonrpc folder. After several tries I haven't got any hanging processes if that folder is renamed/deleted. If it exists often python -m ipykernel_launcher and python -m vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.jupyter_daemon are left in process list.

i have some kind of "test scenario" but it may be difficult to understand:)

UPD: i'll just use systemd to start/stop code-server, this eliminates problem of hanging processes. Though jupyter/runtime folder still gets cluttered with kernel files.

[rchiodo]

Seems like we should clean up the runtime folder kernel.json files. That's likely a result of just killing the process.

[augaug]

Total noob here, but maybe there's something simple to look at in my JSON settings, what I'm observing is closing vscode leaves two conda processes up and running, one of which is zero but the other is consuming ~30% cpu. Basically the entire time I have vscode open there is a conda process at 30% regardless of what I'm doing. Then when I quit vscode and look at task manager it never really quits, those two conda processes are still there, one pegged at ~30%, the other at 0 and a Console Window host also at 0, and vscode is still visible as a running process in task manager, like it just won't quit. Any suggestions?

[rchiodo]

@augaug what is the command line for those processes? That might be conda discovery flaking out.

[augaug]

@rchiodo - how would I find that info? I can provide just not sure what you mean exactly.

One thing I noticed that is slightly odd is that I choose my interpreter, it's a conda env, and it says "('arcgispro-py3-clone':conda)" in status bar of vscode which is correct...but then at some point it just switches, has the same interpreter path etc. and everything works, but no longer says (:conda), it says "Python 3,7,11 64-bit (system)" - I found that odd, not sure if it means anything.

[rchiodo]

@augaug I created a new issue to discuss your problem.