bpo-32604: Expose the subinterpreters C-API in a "private" stdlib module. #1748

ericsnowcurrently · 2017-05-23T04:09:21Z

https://bugs.python.org/issue32604

pitrou · 2017-05-25T20:05:04Z

Doc/library/_interpreters.rst

@@ -0,0 +1,69 @@
+:mod:`_interpreters` --- Low-level interpreters API


There should be a warning about the module being private, and/or the API being subject to change without notice, IMHO.

Good point. For the most part I was following the precedent of the _thread module. However, in this case the module is relatively unstable still and there is no "interpreters" module yet. So a warning makes sense.

Given the not-yet-approved PEP, I think it would be a good idea to use a name that makes it clear that this really is intended as an API for the CPython test suite, rather than for general use.

That would mean going with something like _xxsubinterpreters for now.

Note: one consequence of that is that redistributors that remove the test suite would likely remove this module as well, and I think that's a desirable outcome: we want this as a learning tool for the further development of PEP 554 as a proposal for 3.8 (and to find and fix bugs in the subinterpreter support in 3.7), not as an inadvertent commitment to a particular API.

pitrou · 2017-05-25T20:06:50Z

Lib/test/test__interpreters.py

+import threading
+import _interpreters
+def f():
+    _interpreters.run_string(id, dedent('''


I don't understand where the id variable is coming from... is it the built-in id function?

Good catch. That's a typo that I didn't notice because the resulting exception in the thread does not cause the process to fail.

FYI, adding the missing "id = _interpreters.create()" line causes the test to fail (correctly); the subinterpreter didn't get cleaned up before we reached Py_Finalize(), resulting in Py_FatalError. I'll sort that out separately.

1st1 · 2018-01-20T05:37:37Z

Doc/library/_interpreters.rst

+interpreters and what is unique.  Note particularly that interpreters
+aren't inherently threaded, even though they track and manage Python
+threads.  To run code in an interpreter in a different OS thread, call
+:func:`run_string` in a function that you run in a new Python thread.


Can we call the function run as per the PEP?

Note that the PEP is all about the high-level API we want to expose to users. In contrast, run_string() is the low-level function focused strictly as an analog to PyRun_String*(). As we support other runnable things (e.g. code objects, functions), we will add corresponding low-level functions. The eventual high-level method (i.e. Interpreter.run() will call the appropriate low-level function.

Also, I'm removing the docs for now.

1st1 · 2018-01-20T05:43:53Z

Include/internal/pystate.h

+    // to PyMem_RawFree (the default if not explicitly set to NULL).
+    // The call will happen with the original interpreter activated.
+    void (*free)(void *);
+} _PyCrossInterpreterData;


The actual layout of the struct should go into an "internal" header file:

"Include/pystate.h":

typedef struct _xid _PyCrossInterpreterData;

"Include/internal/pystate.h":

struct _xid { void *data; // etc }

I haven't touched Include/pystate.h. If you are suggesting that I move the typedef (but not the layout) there, I don't see any benefit to that for 3.7. This is not a public API for now.

1st1 · 2018-01-20T05:47:46Z

Modules/_interpretersmodule.c

+    PyThreadState *tstate;
+
+    tstate = PyThreadState_Get();
+    if (tstate == NULL)


You don't need to check for NULL here, PyThreadState_Get aborts if it can't lookup the state.

1st1 · 2018-01-20T05:48:33Z

Modules/_interpretersmodule.c

+    if (id == NULL) {
+        if (PyErr_ExceptionMatches(PyExc_TypeError))
+            PyErr_SetString(PyExc_TypeError, "'id' must be a non-negative int");
+        else


per PEP 7 please use curly braces always :)

1st1 · 2018-01-20T05:50:44Z

Modules/_interpretersmodule.c

+    if (len == 0)
+        return NULL;
+
+    struct _shareditem *shared = PyMem_NEW(struct _shareditem, len+1);


Check if shared is NULL

1st1 · 2018-01-20T05:53:19Z

Modules/_interpretersmodule.c

+    else
+        msg = PyUnicode_FromFormat("%S: %S", exc, value);
+    if (msg == NULL) {
+        err->msg = "unable to format exception";


Redundant? And need to free err?

I'd meant to return err on the next line rather than NULL. :)

ericsnowcurrently · 2018-01-27T19:46:18Z

FYI, I've temporarily removed the Windows support to see if anything else breaks. I'll add it back in soon. I do not plan on merging this without it.

ericsnowcurrently · 2018-01-27T23:22:19Z

@zooba I suspect my problem is with Py_BUILD_CORE. When I don't set it I don't get the symbols out of Include/internal. When I do set it I end up missing stuff from python37.lib at link time. I tried adding a "Link" directive right below where I set Py_BUILD_CORE, but that did not help. :/ Any ideas on what I need to do?

ericsnowcurrently · 2018-01-29T16:17:48Z

@python/windows-team Any of you have ideas on how I can get this PR to link on Windows?

ericsnowcurrently · 2018-01-30T00:58:03Z

In the interest of the feature freeze I'm going to land the PR without Windows support. :/ The risk for beta1 is low since the PR does not have any OS-specific code and the new module is only for use in the test suite. I'll work on getting Windows support done ASAP.

ncoghlan · 2018-01-30T03:37:15Z

Include/internal/pystate.h

+    //
+    // We use the ID rather than the PyInterpreterState to avoid issues
+    // with deleted interpreters.
+    int64_t interp;


Something we'll need to keep an eye on here is not re-using interpreter IDs within the life of a process (otherwise we'll still run into issues with deleted interpreters).

Not that I've ever worked on systems where we introduced bugs by adding a freelist for objects where we assumed IDs would never be re-used or anything ;)

We picked int64_t so that it was both really large and would overflow to negative. When the next ID is negative PyInterpreterState_New() raises RuntimError.

the-knights-who-say-ni added the CLA signed label May 23, 2017

ericsnowcurrently force-pushed the low-level-interpreters-module branch from e0611a8 to bee1baa Compare May 23, 2017 04:38

ericsnowcurrently changed the title ~~Expose the subinterpreters C-API in a "private" stdlib module.~~ bpo-30439: Expose the subinterpreters C-API in a "private" stdlib module. May 23, 2017

ericsnowcurrently force-pushed the low-level-interpreters-module branch from bee1baa to 060a720 Compare May 23, 2017 23:42

ericsnowcurrently mentioned this pull request May 24, 2017

bpo-30439: Add some helpful low-level functions for subinterpreters. #1802

Closed

pitrou reviewed May 25, 2017

View reviewed changes

ericsnowcurrently force-pushed the low-level-interpreters-module branch from 060a720 to 2fd0e64 Compare August 22, 2017 20:01

ericsnowcurrently force-pushed the low-level-interpreters-module branch 2 times, most recently from 114bda4 to 52ca360 Compare September 7, 2017 17:58

Mariatta added needs rebase and removed needs rebase labels Oct 9, 2017

ericsnowcurrently force-pushed the low-level-interpreters-module branch from 52ca360 to 9991d75 Compare November 27, 2017 23:24

ericsnowcurrently force-pushed the low-level-interpreters-module branch from 9991d75 to e9d9b04 Compare December 5, 2017 01:29

ericsnowcurrently force-pushed the low-level-interpreters-module branch from 81de3c8 to bfc025f Compare December 12, 2017 05:12

ericsnowcurrently force-pushed the low-level-interpreters-module branch 2 times, most recently from e05d02e to 396e294 Compare December 23, 2017 00:30

ericsnowcurrently force-pushed the low-level-interpreters-module branch 3 times, most recently from 5266080 to c86647a Compare January 13, 2018 00:49

ericsnowcurrently force-pushed the low-level-interpreters-module branch 2 times, most recently from ff4ae72 to 0981dbf Compare January 20, 2018 01:44

ericsnowcurrently changed the title ~~bpo-30439: Expose the subinterpreters C-API in a "private" stdlib module.~~ bpo-32604: Expose the subinterpreters C-API in a "private" stdlib module. Jan 20, 2018

ericsnowcurrently force-pushed the low-level-interpreters-module branch from 0981dbf to 0be6314 Compare January 20, 2018 02:13

1st1 reviewed Jan 20, 2018

View reviewed changes

ericsnowcurrently added 10 commits January 27, 2018 19:12

Add typedefs.

2df5aff

_shared_exception -> _sharedexception

9bb05cf

Free memory in the error case.

633a3d3

Whitelist instead of blacklist for richcompare ops.

74e7241

Decref in the error case.

bb8619a

Add a Misc/NEWS entry.

d785573

Do not try to print a bytes object.

f856035

Drop some outdated comments.

ba94571

Wrap long lines.

9065001

Fix some compiler warnings.

2ebc68c

ericsnowcurrently force-pushed the low-level-interpreters-module branch from 8ae8507 to 2ebc68c Compare January 27, 2018 19:15

Always use PyMem_NEW().

48291af

ericsnowcurrently force-pushed the low-level-interpreters-module branch 2 times, most recently from ebbf591 to 9718b91 Compare January 27, 2018 21:07

Do not try to allocate too much space.

39df95e

ericsnowcurrently force-pushed the low-level-interpreters-module branch 2 times, most recently from edec009 to 092fccc Compare January 27, 2018 22:50

Add Windows build support.

b07e928

ericsnowcurrently force-pushed the low-level-interpreters-module branch from 092fccc to b07e928 Compare January 27, 2018 23:19

Drop Windows build support (for now).

1bfed0f

ericsnowcurrently merged commit 7f8bfc9 into python:master Jan 30, 2018

ericsnowcurrently mentioned this pull request Jan 30, 2018

bpo-32604: Add the _xxsubinterpreters extension module under Windows. #5436

Closed

ncoghlan reviewed Jan 30, 2018

View reviewed changes

ericsnowcurrently deleted the low-level-interpreters-module branch February 3, 2018 04:54

ericsnowcurrently mentioned this pull request Aug 24, 2018

add low-level module (_xxsubinterpreters) ericsnowcurrently/multi-core-python#1

Closed

ericsnowcurrently mentioned this pull request Apr 10, 2022

[subinterpreters] PEP 554 implementation: add interpreters module #76785

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-32604: Expose the subinterpreters C-API in a "private" stdlib module. #1748

bpo-32604: Expose the subinterpreters C-API in a "private" stdlib module. #1748

ericsnowcurrently commented May 23, 2017 •

edited

Loading

pitrou May 25, 2017

ericsnowcurrently Aug 22, 2017

ncoghlan Jan 20, 2018

ncoghlan Jan 20, 2018

pitrou May 25, 2017

ericsnowcurrently Aug 22, 2017

1st1 Jan 20, 2018

ericsnowcurrently Jan 26, 2018

1st1 Jan 20, 2018

ericsnowcurrently Jan 26, 2018

1st1 Jan 20, 2018

ericsnowcurrently Jan 26, 2018

1st1 Jan 20, 2018

ericsnowcurrently Jan 26, 2018

1st1 Jan 20, 2018

ericsnowcurrently Jan 26, 2018

1st1 Jan 20, 2018

ericsnowcurrently Jan 26, 2018

ericsnowcurrently commented Jan 27, 2018

ericsnowcurrently commented Jan 27, 2018

ericsnowcurrently commented Jan 29, 2018

ericsnowcurrently commented Jan 30, 2018

ncoghlan Jan 30, 2018

ericsnowcurrently Jan 30, 2018

		@@ -0,0 +1,69 @@
		:mod:`_interpreters` --- Low-level interpreters API

bpo-32604: Expose the subinterpreters C-API in a "private" stdlib module. #1748

bpo-32604: Expose the subinterpreters C-API in a "private" stdlib module. #1748

Conversation

ericsnowcurrently commented May 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericsnowcurrently commented Jan 27, 2018

ericsnowcurrently commented Jan 27, 2018

ericsnowcurrently commented Jan 29, 2018

ericsnowcurrently commented Jan 30, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericsnowcurrently commented May 23, 2017 •

edited

Loading