[Python] Fixed a concurrency bug, added begin/endCriticalSection(), a…

…dded UTs

.gitignore

@@ -42,6 +42,6 @@ Thumbs.db
 
 **/doc/html/
 **/dist/
+**/www/
 make*.sh
-email.png
 00Delivery.txt

CHANGELOG.md

@@ -1,3 +1,19 @@
+## v0.3 - XXXX-XX-XX
+
+### New features
+
+* Python: added unit tests and improved the documentation.
+* Python: added functions beginCriticalSection() / endCriticalSection()
+
+### Fixes
+
+* Python: fixed multithreading issue
+
+### Compatibility breakers
+
+*none*
+
+
 ## v0.2 - 2020-05-15
 
 ### New features

README.md

@@ -1,5 +1,5 @@
 **ShlubluLib** is a modular, general purpose, open-source C++ library for Linux and Windows.<br /> 
-The current version is v0.2.
+The current version is v0.3.
 
 This library consists in code I wrote for my own use and that might be useful to others. It is therefore released "as-is", just to be helpful with no 
 warranty of any kind (see license for further details).
@@ -41,7 +41,7 @@ This library currently consists in the following modules:
   * `Debug`: Macros useful for developing and debugging: compilation messages, optimization control, and so on.
   * `Exceptions`: Named exceptions derived from [std::exception](https://www.cplusplus.com/reference/exception/exception/). 
 
-The Doxygen documentation of these modules is available online at [shlublulib.shlublu.org](http://shlublulib.shlublu.org/v0.2/).
+The Doxygen documentation of these modules is available online at [shlublulib.shlublu.org/v0.3/](http://shlublulib.shlublu.org/v0.3/).
 
 Some of these modules require external libraries such as Boost or Python. Such requirements are specified 
 in their documentation.
@@ -60,10 +60,10 @@ it has no standardized replacement so far.
 ## Installation from binaries
 
 Should you just wish to use Shlublulib as a development tool, the binary distribution can be downloaded from our website:
-* [Linux x86 version](http://shlublulib.shlublu.org/dist/shlublulib-linux-v0.2.zip) ([GPG sig](http://shlublulib.shlublu.org/dist/shlublulib-linux-v0.2.zip.asc) - SHA-256: `2b28433de4865ab0b150ecd8951c3e81a27e7dec3900ff6efd4f6eaaf95a9887`)
-* [Win64 x86 version](http://shlublulib.shlublu.org/dist/shlublulib-win64-v0.2.zip) ([GPG sig](http://shlublulib.shlublu.org/dist/shlublulib-win64-v0.2.zip.asc) - SHA-256: `52d0ded07875fdc30e0369acf48eee8ba4d6ee4dae41ce957760ff7126688a3a`)
+* [Linux x86 version](http://shlublulib.shlublu.org/dist/shlublulib-linux-v0.3.zip) ([PGP sig](http://shlublulib.shlublu.org/dist/shlublulib-linux-v0.3.zip.asc) - SHA-256: `4ce0c2cff649297bad791092c6988c9b6023db599efdbf37f949aa284d31dc24`)
+* [Win64 x86 version](http://shlublulib.shlublu.org/dist/shlublulib-win64-v0.3.zip) ([PGP sig](http://shlublulib.shlublu.org/dist/shlublulib-win64-v0.3.zip.asc) - SHA-256: `2ce011a85a25b4b2fe3698318be82fb89fb4a08326eb766f9a1537a82de91420`)
 
-These archives contain:
+These archives are signed with [my PGP key](https://keyserver.ubuntu.com/pks/lookup?search=shlublu%40yahoo.fr&fingerprint=on&op=index) and contain:
 * the library file to link to your client programs
 * the `include` directory to add to your include path 
 * the license file "EUPL LICENCE.txt" applicable to this library

doxygen.conf

@@ -38,7 +38,7 @@ PROJECT_NAME           = ShlubluLib
 # could be handy for archiving the generated documentation or if some version
 # control system is used.
 
-PROJECT_NUMBER = v0.2
+PROJECT_NUMBER = v0.3
 
 # Using the PROJECT_BRIEF tag one can provide an optional one line description
 # for a project that appears at the top of each page and should give viewer a

include/shlublu/binding/Python.h

@@ -24,8 +24,6 @@ namespace shlublu
 	users to use advanced features of the official API when needed. In particular, this module focuses of making the references count handling simpler
 	and less error-prone than doing it manually as CPython requires.
 
-	@remark All functions of this namespace are thread safe. They share a MutexLock to ensure this.
-
 	<b>Requirements</b><br />
 	This module uses the official `Python3.x` library.
 	Client programs need their include path, libraries path and linker input to be updated as follows:
@@ -155,6 +153,20 @@ namespace shlublu
 	</li>
 	<li>More advanced features are explained below.</li>
 	</ul>
+
+	@attention <b>About concurrency</b><br/>
+	The Python interpreter is unique and shared by all the threads of the process, *and so is its memory state*, including imports and objects.<br /> 
+	All functions below support concurrent access thanks to the use	of a MutexLock. However, groups of Python calls that have a dependency
+	relationship in your C++ code should be surrounded by `beginCriticalSection()` / `endCriticalSection()`	to prevent any concurrent call
+	to the global interpreter to occur in the middle of the section they constitute. Such concurrent accesses would lead the interpreter to crash.<br />
+	Should you need more isolation, you can use the CPython API to setup
+	<a href="https://docs.python.org/3/c-api/init.html#c.Py_NewInterpreter">multiple interpreters</a> knowing that there are caveats
+	described in the documentation. One of this limitation is <a href="https://docs.python.org/3/c-api/init.html#non-python-created-threads">the use of 
+	the `PyGILState_*()` API</a>. shlublu::Python doesn't use this API so there is no issue in that respect.<br />
+	Forking is ok as the child process receives a full copy of the memory of the parent process made at `fork()` time.
+	Testing shows the interpreter supports this very well, providing a full isolation, as long as the fork operation
+	is conducted from a mono-thread process. Should you wish to fork a multi-threads process
+	<a href="https://docs.python.org/3/c-api/init.html#cautions-about-fork">there are some cautions</a>.
 */
 namespace Python
 {
@@ -249,6 +261,80 @@ namespace Python
 	*/
 	void execute(Program const& program);
 
+
+	/**
+		Prevent other threads to access to the global interpreter.
+
+		Using this function is relevant in multi-threaded applications when groups of Python calls have a dependency relationships: 
+		concurrent calls to the interpreter occuring in the middle of the execution of such groups would lead the interpreter to crash.
+
+		Calls to this function should be followed in the same thread by as many calls to endCriticalSection(). Not doing so is
+		a cause of deadlocks.
+
+		@exception BindingException if Python is not initialized.
+
+		<b>Example</b>
+		@code
+		Python::init("pythonBinding");
+
+		Python::execute(Python::Program({ "def inc(x):", "\treturn x + 1" }));
+
+		const long nbIt(5000000L);
+		long x(0);
+		long y(0);
+
+		const auto job
+		(
+			[](long iterations) -> long
+			{
+				long v(0);
+				const auto inc(Python::callable(Python::moduleMain, "inc"));
+
+				for (long i = 0; i < iterations; ++i)
+				{
+					Python::beginCriticalSection();
+					const auto pyVal(Python::call(inc, Python::arguments(1, PyLong_FromLong(v))));
+
+					v = PyLong_AsLong(pyVal);
+
+					Python::forgetArgument(pyVal);
+					Python::endCriticalSection();
+				}
+
+				return v;
+			}
+		);
+
+		std::thread tx
+		(
+			[&x, &job, nbIt]()
+			{
+				x = job(nbIt);
+			}
+		);
+
+		y = job(nbIt);
+
+		tx.join();
+
+		// Assert: x == y == nbIt
+
+		Python::shutdown();
+		@endcode
+	*/
+	void beginCriticalSection();
+
+
+	/**
+		Allow other threads to access to the global interpreter.
+
+		Calls to this function should match calls to beginCriticalSection() performed in the same thread.
+
+		@exception BindingException if this call doesn't match a call to beginCriticalSection() performed in the same thread.
+	*/
+	void endCriticalSection();
+
+
 	//////////
 	// The returned values below are garbage collected automatically, though calling forgetArgument() is possible ahead.
 

src/binding/Python.cpp

@@ -104,6 +104,11 @@ void Python::init(std::string const& programName, PathEntriesList const& pathLis
 
         Py_SetProgramName(__pythonArgv0);
         Py_Initialize();
+
+        if (!PyEval_ThreadsInitialized())
+        {
+            PyEval_InitThreads();
+        }
 
         atexit(shutdown);
 
@@ -187,6 +192,18 @@ void Python::execute(Program const& program)
 }
 
 
+void Python::beginCriticalSection()
+{
+    __pythonShouldBeInitialized();
+}
+
+
+void Python::endCriticalSection()
+{
+    __pythonRelease();
+}
+
+
 Python::ScopeRef Python::import(std::string const& moduleName)
 {
     __pythonShouldBeInitialized();

tests/binding/TestPython.cpp

@@ -2,6 +2,8 @@
 
 #include "CppUnitTest.h"
 
+#include <thread>
+
 #include <shlublu/binding/Python.h>
 #include <shlublu/util/Debug.h>
 SHLUBLU_OPTIMIZE_OFF();
@@ -795,5 +797,88 @@ namespace binding_Python
 			Python::shutdown();
 		}
 	};
+
+	TEST_CLASS(threadingTest)
+	{
+		TEST_METHOD(executeIsThreadSafe)
+		{
+			std::thread t1
+			(
+				[]()
+				{
+					Python::init("Thread1");
+					Python::execute(Python::Program({ "sum1 = 0", "for i in range(0, 10000000):", "\tsum1 += 1" }));
+				}
+			);
+
+			std::thread t2
+			(
+				[]()
+				{
+					Python::init("Thread2");
+					Python::execute(Python::Program({ "sum2 = 0", "for i in range(0, 10000000):", "\tsum2 += 1" }));
+				}
+			);
+
+			t1.join();
+			t2.join();
+
+			Assert::AreEqual(PyLong_AsLong(Python::object(Python::moduleMain, "sum1")), PyLong_AsLong(Python::object(Python::moduleMain, "sum2")));
+			Assert::AreEqual(10000000L, PyLong_AsLong(Python::object(Python::moduleMain, "sum1")));
+
+			Python::shutdown();
+		}
+
+
+		TEST_METHOD(transactionsAreThreadSafe)
+		{
+			Python::init("pythonBinding");
+
+			Python::execute(Python::Program({ "def inc(x):", "\treturn x + 1" }));
+
+			const long nbIt(5000000L);
+			long x(0);
+			long y(0);
+
+			const auto job
+			(
+				[](long iterations) -> long
+				{
+					long v(0);
+					const auto inc(Python::callable(Python::moduleMain, "inc"));
+
+					for (long i = 0; i < iterations; ++i)
+					{
+						Python::beginCriticalSection();
+						const auto pyVal(Python::call(inc, Python::arguments(1, PyLong_FromLong(v))));
+
+						v = PyLong_AsLong(pyVal); 
+
+						Python::forgetArgument(pyVal);
+						Python::endCriticalSection();
+					}
+
+					return v;
+				} 
+			);
+
+			std::thread tx
+			(
+				[&x, &job, nbIt]()
+				{
+					x = job(nbIt);
+				}
+			);
+
+			y = job(nbIt);
+
+			tx.join();
+
+			Assert::AreEqual(x, y);
+			Assert::AreEqual(nbIt, x);
+
+			Python::shutdown();
+		}
+	};
 }