Skip to content

Commit

Permalink
fixed the python memory leak (facebookresearch#844)
Browse files Browse the repository at this point in the history
Summary:
related to the issue facebookresearch#841
Pull Request resolved: facebookresearch#844

Test Plan:
Detect leaks with valgrind and python compiled in debug mode; with docker and ubuntu 16.04;

`docker run -it -v ~/development/otherstuff/dockerlinux/:/home/  ubuntu:16.04 bash`
```
apt update
apt upgrade
apt-get install g++ valgrind -y
apt-get install libssl-dev zlib1g-dev   # needed by ssl and zlib enabled python (in order to run get-pip.py and get pip)
```

Download python's source code.
Do these modifications :
 - Uncomment Py_USING_MEMORY_DEBUGGER in Objects/obmalloc.c
 - Uncomment the lines in Misc/valgrind-python.supp that suppress the warnings for PyObject_Free and PyObject_Realloc
 - Uncomment ssl part in Modules/Setup.dist:
```
SSL=/usr
_ssl _ssl.c \
        -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
        -L$(SSL)/lib -lssl -lcrypto

```

now compile Python with
```
./configure --with-pydebug --without-pymalloc --with-valgrind --prefix /opt/debugpython/ OPT=-g
make install
```

get `get-pip.py` from https://bootstrap.pypa.io/get-pip.py
run `/opt/debugpython/bin/python get-pip.py`

Now you have custom Python with pip.
Install fasttext.

Run valgrind on this custom built Python :
```
valgrind --tool=memcheck --leak-check=full --suppressions=/root/Python-2.7.8/Misc/valgrind-python.supp /opt/debugpython/bin/python memleak.py
```

With memleak.py :
```
#!/usr/bin/env python

import fasttext

ft = fasttext.load_model("dbpedia.bin")

def leaker():
    for i in range(1000000):
        _ = ft.predict("hello, world!")

if __name__ == '__main__':
    leaker()
```

 ------------

References:
 - https://stackoverflow.com/questions/5937337/building-python-with-ssl-support-in-non-standard-location
 - https://www.1stbyte.com/2005/06/26/configure-and-compile-python-with-zlib/
 - https://askubuntu.com/questions/597906/how-to-install-pip-and-a-python-package-for-self-installed-python
 - https://stackoverflow.com/questions/20112989/how-to-use-valgrind-with-python

Reviewed By: EdouardGrave

Differential Revision: D16161994

Pulled By: Celebio

fbshipit-source-id: 31995d810a15a64139670abeff48f0e8e1ad5968
  • Loading branch information
lukeandshuo authored and facebook-github-bot committed Jul 16, 2019
1 parent b5b7d30 commit 40a7744
Showing 1 changed file with 7 additions and 2 deletions.
9 changes: 7 additions & 2 deletions python/fasttext_module/fasttext/pybind/fasttext_pybind.cc
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,15 @@ py::str castToPythonString(const std::string& s, const char* onUnicodeError) {
// for python 2 and not encoded for python 3 :
// https://github.com/pybind/pybind11/blob/ccbe68b084806dece5863437a7dc93de20bd9b15/include/pybind11/pytypes.h#L930
#if PY_MAJOR_VERSION < 3
handle = PyUnicode_AsEncodedString(handle, "utf-8", onUnicodeError);
PyObject* handle_encoded =
PyUnicode_AsEncodedString(handle, "utf-8", onUnicodeError);
Py_DECREF(handle);
handle = handle_encoded;
#endif

return py::str(handle);
py::str handle_str = py::str(handle);
Py_DECREF(handle);
return handle_str;
}

std::pair<std::vector<py::str>, std::vector<py::str>> getLineText(
Expand Down

0 comments on commit 40a7744

Please sign in to comment.