High memory usage

[gramster]

@suiahaw commented on Tue Mar 26 2019

Issue Type: Bug

Some questions about the python plugin encountered when building python library indexes

I strongly hope that the python plugin does not read the information into memory in real time when creating the python library index, but instead saves the index file in order to speed up the time and reduce memory overhead.

The python library is really too big. Sometimes I have to wait a few minutes for writing a small amount of code.

Extension version: 2019.3.6139
VS Code version: Code 1.32.3 (a3db5be9b5c6ba46bb7555ec5d60178ecc2eaae4, 2019-03-14T23:43:35.476Z)
OS version: Windows_NT x64 10.0.17763

System Info

Item	Value
CPUs	Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz (4 x 2712)
GPU Status	2d_canvas: enabled checker_imaging: disabled_off flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on native_gpu_memory_buffers: disabled_software rasterization: enabled surface_synchronization: enabled_on video_decode: enabled webgl: enabled webgl2: enabled
Memory (System)	15.86GB (8.60GB free)
Process Argv
Screen Reader	no
VM	46%

---

@algo99 commented on Wed Mar 27 2019

I can confirm, on Windows 10.

After update (yesterday) of Python Language Server and/or vscode_python extension VSCode not more usable. See the disk I/O and memory consumption:

The extension proceeds after VSCode start:

Analyzing in background, 6879 items left ...

and this process does not stop.
I tried several times to restart VSCode, but it does not help.

---

@algo99 commented on Wed Mar 27 2019

Just noted in Python output pane:

...
[Info  - 09:17:10] Microsoft Python Language Server version 0.2.31.0
[Info  - 09:17:10] Initializing for c:\3rd\WinPython\python-3.7.2.amd64\python.exe
Traceback (most recent call last):
  File "C:\Users\zkr\.vscode\extensions\ms-python.python-2019.3.6139\languageServer.0.2.31\scrape_module.py", line 1497, in <module>
    state.collect_second_level_members()
  File "C:\Users\zkr\.vscode\extensions\ms-python.python-2019.3.6139\languageServer.0.2.31\scrape_module.py", line 944, in collect_second_level_members
    self._collect_members(mi.value, mi.members, substitutes, mi)
  File "C:\Users\zkr\.vscode\extensions\ms-python.python-2019.3.6139\languageServer.0.2.31\scrape_module.py", line 958, in _collect_members
    raise RuntimeError("failed to import module")
RuntimeError: failed to import module
Traceback (most recent call last):
  File "C:\Users\zkr\.vscode\extensions\ms-python.python-2019.3.6139\languageServer.0.2.31\scrape_module.py", line 1497, in <module>
    state.collect_second_level_members()
  File "C:\Users\zkr\.vscode\extensions\ms-python.python-2019.3.6139\languageServer.0.2.31\scrape_module.py", line 944, in collect_second_level_members
    self._collect_members(mi.value, mi.members, substitutes, mi)
  File "C:\Users\zkr\.vscode\extensions\ms-python.python-2019.3.6139\languageServer.0.2.31\scrape_module.py", line 958, in _collect_members
    raise RuntimeError("failed to import module")
RuntimeError: failed to import module
Traceback (most recent call last):
  File "C:\Users\zkr\.vscode\extensions\ms-python.python-2019.3.6139\languageServer.0.2.31\scrape_module.py", line 1489, in <module>
    state.initial_import(sys.argv[2])
  File "C:\Users\zkr\.vscode\extensions\ms-python.python-2019.3.6139\languageServer.0.2.31\scrape_module.py", line 872, in initial_import
    mod = __import__(self.module_name)
  File "c:\3rd\winpython\python-3.7.2.amd64\lib\site-packages\cvxopt\__init__.py", line 50, in <module>
    import cvxopt.base
ImportError: DLL load failed: Das angegebene Modul wurde nicht gefunden.

---

@dilzeem commented on Wed Mar 27 2019

I am also getting the same issue as well on Windows 10.

It only happens once I start debugging some code.
It renders VScode unusable.

---

@suiahaw commented on Wed Mar 27 2019

It only happens when the python plugin loads the python library, which requires a lot of CPU and memory. And these information that is loaded into memory is estimated to be used for IntelliCode.

---

@ray306 commented on Wed Mar 27 2019

The same problem happens today. I think it's a bug of new Python plugin just released yesterday (March 26).

---

@dilzeem commented on Wed Mar 27 2019

Possibly a duplicate of of this in the python language server:

#450

---

@suiahaw commented on Wed Mar 27 2019

Possibly a duplicate of of this in the python language server:

Microsoft/python-language-server#450

I also think that the issues area of the python plugin has a large part of similar content related to such a bug. It seems that this bug is very disturbing programming and needs to be solved urgently. ;-)

[MikhailArkhipov]

Could you post specific memory consumption and approx time time to when analysis completes. Generally you don't have to wait until it completes as it is a background task that collects information in stages. 4GB consumption may happen, eventually memory is released.

#450 was about 30GB+ or runaway consuming all memory, so it is not a duplicate. Number of tasks is limited so process should not be consuming all CPU, limit is ~40% or so.

If we can clone projects for investigation it would be helpful too.

Thanks.

[algo99]

It was strange enough for me.

1. I work normally with WinPython. As I wrote above the analysis task do not stop (also over night). At the end VSCode just crashed. All the time the GUI was not usable at all.

2. Today I installed Anaconda3, and then tried again, and VSCode works with Python as usual, just these messages in output Python pane still here:

Traceback (most recent call last):
  File "C:\Users\zkr\.vscode\extensions\ms-python.python-2019.3.6139\languageServer.0.2.31\scrape_module.py", line 1489, in <module>
    state.initial_import(sys.argv[2])
  File "C:\Users\zkr\.vscode\extensions\ms-python.python-2019.3.6139\languageServer.0.2.31\scrape_module.py", line 872, in initial_import
    mod = __import__(self.module_name)
  File "c:\3rd\anaconda3\lib\site-packages\scipy\__init__.py", line 62, in <module>
    from numpy import show_config as show_numpy_config
  File "c:\3rd\anaconda3\lib\site-packages\numpy\__init__.py", line 140, in <module>
    from . import _distributor_init
  File "c:\3rd\anaconda3\lib\site-packages\numpy\_distributor_init.py", line 34, in <module>
    from . import _mklinit
ImportError: DLL load failed: Das angegebene Modul wurde nicht gefunden.

Looks like the server can not import some DLL related to numpy now.
When you look at my report above (with WinPython), there was some DLL related to cvxpy.

Could it be some ABI issue ?
Because I noted yesterday, that I updated my Visual Studio Build Tools 2017 and decided reinstall cvxpy package (actually rebuilding related DLLs)

pip install -U cvxpy

And after restarting VSCode (yesterday with WinPython) there was not messages with cvxpy, but many related to another packages like "DLL load failed"

[dilzeem]

Some further information:

I did have consumption at 22GB+, with CPU usage at around 15-25%.

It happens when I open any python file. This is after 30 seconds after opening a simple python file, which just imports pandas.

It was analyzing a few files indicated at the bottom of the toolbar:

After a few minutes it has stabilized to this:

I believe the issue is when I have virtual environments folder within the workspace, it causes a huge spike in RAM usage by analyzing over 7000+ files.

Version: 1.32.3 (user setup)
Commit: a3db5be9b5c6ba46bb7555ec5d60178ecc2eaae4
Date: 2019-03-14T23:43:35.476Z
Electron: 3.1.6
Chrome: 66.0.3359.181
Node.js: 10.2.0
V8: 6.6.346.32
OS: Windows_NT x64 10.0.17134

[suiahae]

After analysis, I think memory consumption
is caused when the python plugin creats the python library index by reading the information into memory in real time. (The memory consumed by the Python library index is for the IntelliSense, I think)

Further, memory consumption and approx time time to when analysis completes depends on the size of the extra library installed for Python. (In other words, the bigger the Python library, the greater the memory occupies, and the more time required)
For anaconda installed by default, memory consumption is about 6GB and approx time time to when analysis completes is about 15 mins.

I can debug existing code before it's done, but I can't use IntelliSense when writing code.

Thanks！

[ly-cultureiq]

RAM consumption for me hit 7.7GB before running out of memory.
Amount of modules remaining to analyze continuously rises as well

[jakebailey]

@algo99 The DLL importing thing was technically a problem in the old version, but just wasn't printed. The currently released version does print it, but we merged a PR which when released will re-hide it (see #823).

@suiahaw The language server doesn't keep an on-disk version of the analysis yet (and never has, apart from scraped information from compiled libraries that cannot be directly parsed and analyzed). That is #472.

[gramster]

@dilzeem, import pandas means the LS will analyze pandas and its transitive dependencies (numpy etc), so there is a lot going on at first. Taking a couple of minutes for that is not unexpected, but after that you should have many completions available (and many which we simply did not have at all in the previous language server). Just want to set expectations around expected behavior versus actual bugs. We want to track down bugs like memory leaks and analysis not completing, and we will make improvements in the future around caching and type stubs to speed up analysis of common packages, but its also the case that having some initial cost to analysis is to be expected when importing complex packages.

[suiahae]

@algo99 The DLL importing thing was technically a problem in the old version, but just wasn't printed. The currently released version does print it, but we merged a PR which when released will re-hide it (see #823).

@suiahaw The language server doesn't keep an on-disk version of the analysis yet (and never has, apart from scraped information from compiled libraries that cannot be directly parsed and analyzed). That is #472.

If no an on-disk version of the analysis exists, this means that whenever I write a Python program using vscode, I would have to suffer extra memory and time overhead. This will make me unable to use vscode as my main development tool. an on-disk version of the analysis is necessary. On the other hand, the extra disk overhead brought by the analysis can be solved by database compression and periodic reminding users to clean up.

[MikhailArkhipov]

@dilzeem - 6GB peak is not unusual at the moment considering that used to be 30GB+ in #450, 20 appears excessive though.

[dilzeem]

@MikhailArkhipov, @gramster thanks for the quick response.

Okay, it was surprising as I didn't have any issues like this until the March Python Extension update. So it was unexpected behavior.

Okay I will see if I can recreate the 25GB+ scenario tomorrow, and give you an update. The project I was using had a lot of modules, mainly scikit-learn, pandas, dash and gensim.

The computer actually crashes when I opened two similar projects in different windows.

[micw523]

I'm not sure if this is an old bug, since I'm also seeing this behavior only after the most recent update.

[ranka47]

I started facing this issue in the last week. My laptop has 8GB RAM and the OS is Ubuntu 16.04. After opening a python file, the RAM usage shoots up until it is completely eaten. The only way I am able to use VSCode is by disabling the extension.
I tried disabling python.jediEnabled and tried removing it from .vscode folder and reinstalling it. However, the issue still persists.

[dilzeem]

Just an update:

Just importing 4 libraries. Initially got this with analyzing in background at the bottom which is frozen with 9709 items left:

After it has been analyzing for approx 30 mins it is still 20+ GB Ram Usage. Also the number of items left has increased to 10479.

After 40 mins of analyzing VScode crashed.