Excessive memory use when caching large packages

[wolever]

I'm getting a MemoryError while trying to install a large package (matplotlib) in a low memory (512mb) environment. It appears that the cause is the caching mechanism, as disabling the cache fixes the issue.

$ pip --version
pip 7.1.0 from /…/virtualenv/local/lib/python2.7/site-packages (python 2.7)
$ pip install matplotlib==1.4.3
Collecting matplotlib==1.4.3
  Downloading matplotlib-1.4.3.tar.gz (50.4MB)
    99% |████████████████████████████████| 50.4MB 20.5MB/s eta 0:00:01
Exception:
Traceback (most recent call last):
  File ".../pip/basecommand.py", line 223, in main
    status = self.run(options, args)
  File ".../pip/commands/install.py", line 282, in run
    requirement_set.prepare_files(finder)
  File ".../pip/req/req_set.py", line 334, in prepare_files
    functools.partial(self._prepare_file, finder))
  File ".../pip/req/req_set.py", line 321, in _walk_req_to_install
    more_reqs = handler(req_to_install)
  File ".../pip/req/req_set.py", line 491, in _prepare_file
    session=self.session)
  File ".../pip/download.py", line 825, in unpack_url
    session,
  File ".../pip/download.py", line 673, in unpack_http_url
    from_path, content_type = _download_http_url(link, session, temp_dir)
  File ".../pip/download.py", line 886, in _download_http_url
    _download_url(resp, link, content_file)
  File ".../pip/download.py", line 621, in _download_url
    for chunk in progress_indicator(resp_read(4096), 4096):
  File ".../pip/utils/ui.py", line 133, in iter
    for x in it:
  File ".../pip/download.py", line 586, in resp_read
    decode_content=False):
  File ".../pip/_vendor/requests/packages/urllib3/response.py", line 307, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File ".../pip/_vendor/requests/packages/urllib3/response.py", line 243, in read
    data = self._fp.read(amt)
  File ".../pip/_vendor/cachecontrol/filewrapper.py", line 54, in read
    self.__callback(self.__buf.getvalue())
  File ".../pip/_vendor/cachecontrol/controller.py", line 224, in cache_response
    self.serializer.dumps(request, response, body=body),
  File ".../pip/_vendor/cachecontrol/serialize.py", line 81, in dumps
    ).encode("utf8"),
MemoryError
$ pip install --no-cache-dir matplotlib==1.4.3
…
Successfully installed matplotlib-1.4.3

[fbender]

Experiencing the same with basemap on a 4GB machine (Win 7):

$>pip install https://github.com/matplotlib/base
map/archive/v1.0.7rel.tar.gz
Collecting https://github.com/matplotlib/basemap/archive/v1
.0.7rel.tar.gz
  Downloading https://github.com/matplotlib/basemap/archive
/v1.0.7rel.tar.gz
     \ 131.5MB 1.3MB/sException:
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\pip\basecommand.py",
line 215, in main
    status = self.run(options, args)
  File "c:\python27\lib\site-packages\pip\commands\install.
py", line 299, in run
    requirement_set.prepare_files(finder)
  File "c:\python27\lib\site-packages\pip\req\req_set.py",
line 370, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "c:\python27\lib\site-packages\pip\req\req_set.py",
line 587, in _prepare_file
    session=self.session, hashes=hashes)
  File "c:\python27\lib\site-packages\pip\download.py", lin
e 810, in unpack_url
    hashes=hashes
  File "c:\python27\lib\site-packages\pip\download.py", lin
e 649, in unpack_http_url
    hashes)
  File "c:\python27\lib\site-packages\pip\download.py", lin
e 871, in _download_http_url
    _download_url(resp, link, content_file, hashes)
  File "c:\python27\lib\site-packages\pip\download.py", lin
e 597, in _download_url
    consume(downloaded_chunks)
  File "c:\python27\lib\site-packages\pip\utils\__init__.py
", line 860, in consume
    deque(iterator, maxlen=0)
  File "c:\python27\lib\site-packages\pip\download.py", lin
e 563, in written_chunks
    for chunk in chunks:
  File "c:\python27\lib\site-packages\pip\utils\ui.py", lin
e 139, in iter
    for x in it:
  File "c:\python27\lib\site-packages\pip\download.py", lin
e 552, in resp_read
    decode_content=False):
  File "c:\python27\lib\site-packages\pip\_vendor\requests\
packages\urllib3\response.py", line 353, in stream
    data = self.read(amt=amt, decode_content=decode_content
)
  File "c:\python27\lib\site-packages\pip\_vendor\requests\
packages\urllib3\response.py", line 310, in read
    data = self._fp.read(amt)
  File "c:\python27\lib\site-packages\pip\_vendor\cachecont
rol\filewrapper.py", line 54, in read
    self.__callback(self.__buf.getvalue())
  File "c:\python27\lib\site-packages\pip\_vendor\cachecont
rol\controller.py", line 275, in cache_response
    self.serializer.dumps(request, response, body=body),
  File "c:\python27\lib\site-packages\pip\_vendor\cachecont
rol\serialize.py", line 87, in dumps
    ).encode("utf8"),
MemoryError

[renekliment]

I can confirm this. We had this problem over at AlexaPi when installing pocketsphinx. Using the workaround for now.

[xavfernandez]

This might be less relevant with the switch from json to msgpack in psf/cachecontrol#115 (which should be less memory consuming):

(tmp-c57a6e553d812e0) [xfernandez@xaferiba tmp-c57a6e553d812e0]$ pip -V
pip 9.0.1 from /home/xfernandez/.virtualenvs/tmp-c57a6e553d812e0/lib/python3.6/site-packages (python 3.6)
(tmp-c57a6e553d812e0) [xfernandez@xaferiba tmp-c57a6e553d812e0]$ /usr/bin/time -v pip download matplotlib --no-deps
Collecting matplotlib
  Downloading https://some.pypi/root/pypi/+f/500/8fb7ea1062dc9/matplotlib-2.0.0-1-cp36-cp36m-manylinux1_x86_64.whl (14.7MB)
    100% |████████████████████████████████| 14.7MB 108kB/s
  Saved ./matplotlib-2.0.0-1-cp36-cp36m-manylinux1_x86_64.whl
Successfully downloaded matplotlib
        Command being timed: "pip download matplotlib --no-deps"
        ...
        Maximum resident set size (kbytes): 104924
        ...

# Remove cache, downloaded file and install pip on master

(tmp-c57a6e553d812e0) [xfernandez@xaferiba tmp-c57a6e553d812e0]$ pip -V
pip 10.0.0.dev0 from /home/xfernandez/.virtualenvs/tmp-c57a6e553d812e0/lib/python3.6/site-packages (python 3.6)
(tmp-c57a6e553d812e0) [xfernandez@xaferiba tmp-c57a6e553d812e0]$ /usr/bin/time -v pip download matplotlib --no-deps
Collecting matplotlib
  Downloading https://some.pypi/root/pypi/+f/500/8fb7ea1062dc9/matplotlib-2.0.0-1-cp36-cp36m-manylinux1_x86_64.whl (14.7MB)
    100% |████████████████████████████████| 14.7MB 3.5MB/s
  Saved ./matplotlib-2.0.0-1-cp36-cp36m-manylinux1_x86_64.whl
Successfully downloaded matplotlib
        Command being timed: "pip download matplotlib --no-deps"
        ...
        Maximum resident set size (kbytes): 80384
        ...

So I guess this won't solve the issue.

The next possible thing would be to create an issue for cachecontrol:
it could catch the MemoryError, log a warning, give up on caching the response but still provide the response to requests/pip.

[pradyunsg]

Does anyone know what the state of this is? Was a PR made of cachecontrol at some point or not? Is there something that pip can do completely on it's own about this?