Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fixed the python memory leak (facebookresearch#844)
Summary: related to the issue facebookresearch#841 Pull Request resolved: facebookresearch#844 Test Plan: Detect leaks with valgrind and python compiled in debug mode; with docker and ubuntu 16.04; `docker run -it -v ~/development/otherstuff/dockerlinux/:/home/ ubuntu:16.04 bash` ``` apt update apt upgrade apt-get install g++ valgrind -y apt-get install libssl-dev zlib1g-dev # needed by ssl and zlib enabled python (in order to run get-pip.py and get pip) ``` Download python's source code. Do these modifications : - Uncomment Py_USING_MEMORY_DEBUGGER in Objects/obmalloc.c - Uncomment the lines in Misc/valgrind-python.supp that suppress the warnings for PyObject_Free and PyObject_Realloc - Uncomment ssl part in Modules/Setup.dist: ``` SSL=/usr _ssl _ssl.c \ -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \ -L$(SSL)/lib -lssl -lcrypto ``` now compile Python with ``` ./configure --with-pydebug --without-pymalloc --with-valgrind --prefix /opt/debugpython/ OPT=-g make install ``` get `get-pip.py` from https://bootstrap.pypa.io/get-pip.py run `/opt/debugpython/bin/python get-pip.py` Now you have custom Python with pip. Install fasttext. Run valgrind on this custom built Python : ``` valgrind --tool=memcheck --leak-check=full --suppressions=/root/Python-2.7.8/Misc/valgrind-python.supp /opt/debugpython/bin/python memleak.py ``` With memleak.py : ``` #!/usr/bin/env python import fasttext ft = fasttext.load_model("dbpedia.bin") def leaker(): for i in range(1000000): _ = ft.predict("hello, world!") if __name__ == '__main__': leaker() ``` ------------ References: - https://stackoverflow.com/questions/5937337/building-python-with-ssl-support-in-non-standard-location - https://www.1stbyte.com/2005/06/26/configure-and-compile-python-with-zlib/ - https://askubuntu.com/questions/597906/how-to-install-pip-and-a-python-package-for-self-installed-python - https://stackoverflow.com/questions/20112989/how-to-use-valgrind-with-python Reviewed By: EdouardGrave Differential Revision: D16161994 Pulled By: Celebio fbshipit-source-id: 31995d810a15a64139670abeff48f0e8e1ad5968
- Loading branch information