[subinterpreters] PEP 554 implementation: add interpreters module

[ericsnowcurrently]

BPO	32604
Nosy	@ncoghlan, @abalkin, @pitrou, @vstinner, @pmp-p, @ericsnowcurrently, @zware, @zooba, @applio, @emilyemorehouse, @pablogsal, @miss-islington, @nanjekyejoannah
PRs	bpo-32604: Expose the subinterpreters C-API in a "private" stdlib module. #1748 bpo-32604: Add the _xxsubinterpreters extension module under Windows. #5436 bpo-32604: NULL-terminate kwlist in channel_drop_interpreter(). #5437 bpo-32604: Fix memory leaks in the new _xxsubinterpreters module. #5507 [3.7] bpo-32604: Fix memory leaks in the new _xxsubinterpreters module. (GH-5507) #5509 bpo-32604: Make _xxsubinterpreters build on Windows #5516 [3.7] bpo-32604: Make _xxsubinterpreters build on Windows (GH-5516) #5624 bpo-32604: Clean up created subinterpreters before runtime finalization. #5709 [3.7] bpo-32604: Clean up created subinterpreters before runtime finalization. (gh-5709) #5710 bpo-32604: Swap threads only if the interpreter is different. #5778 [3.7] bpo-32604: Swap threads only if the interpreter is different. (gh-5778) #5783 bpo-32604: Remove xid registry. #6813 bpo-32604: Improve subinterpreter tests. #6914 bpo-32604: Implement force-closing channels. #6937 bpo-33615: avoid extra decref #7251 bpo-33615: Temporarily disable a test that is triggering crashes on a few buildbots. #7288 bpo-33724: Use the right format code for int64_t in subinterpreters code. #7330 bpo-32604: [_xxsubinterpreters] Propagate exceptions. #19768 bpo-32604: Add support for a "default" arg in channel_recv(). #19770 bpo-32604: [_xxsubinterpreters] Add channel_send_wait(). #19829 Revert "bpo-32604: [_xxsubinterpreters] Propagate exceptions. (GH-19768)" #20089 gh-76785: Multiple Interpreters in the Stdlib (PEP 554) #18817 bpo-32604: PEP 554 for use in test suite #19985 Revert "bpo-32604: PEP 554 for use in test suite (GH-19985)" #20465 bpo-32604: Fix reference leak in select module #20600 bpo-32604: Recommit "bpo-32604: PEP 554 for use in test suite (GH-19985)" #20611 bpo-32604: Disable failing tests #20777 bpo-32604: Clean up test.support.interpreters. #20926

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericsnowcurrently'
closed_at = None
created_at = <Date 2018-01-20.01:58:01.942>
labels = ['expert-subinterpreters', 'type-feature', '3.10', 'docs']
title = '[subinterpreters] PEP 554 implementation: add interpreters module'
updated_at = <Date 2020-06-17.00:42:48.323>
user = 'https://github.com/ericsnowcurrently'

bugs.python.org fields:

activity = <Date 2020-06-17.00:42:48.323>
actor = 'eric.snow'
assignee = 'eric.snow'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'Subinterpreters']
creation = <Date 2018-01-20.01:58:01.942>
creator = 'eric.snow'
dependencies = []
files = []
hgrepos = []
issue_num = 32604
keywords = ['patch']
message_count = 66.0
messages = ['310314', '310316', '310317', '310318', '310448', '310562', '311210', '311211', '311212', '311217', '311218', '311219', '311220', '311222', '311224', '311299', '311533', '311534', '311536', '311538', '312001', '312004', '312260', '312262', '312288', '312362', '312367', '312368', '312369', '312370', '312372', '312443', '312447', '312837', '312888', '312975', '312976', '312979', '313010', '316657', '316717', '316850', '316936', '318333', '318479', '319560', '367573', '368349', '368395', '368846', '368849', '368851', '368899', '368902', '369378', '369454', '369455', '370130', '370140', '370631', '370660', '371157', '371173', '371176', '371704', '371707']
nosy_count = 13.0
nosy_names = ['ncoghlan', 'belopolsky', 'pitrou', 'vstinner', 'pmpp', 'eric.snow', 'zach.ware', 'steve.dower', 'davin', 'emilyemorehouse', 'pablogsal', 'miss-islington', 'nanjekyejoannah']
pr_nums = ['1748', '5436', '5437', '5507', '5509', '5516', '5624', '5709', '5710', '5778', '5783', '6813', '6914', '6937', '7251', '7288', '7330', '19768', '19770', '19829', '20089', '18817', '19985', '20465', '20600', '20611', '20777', '20926']
priority = None
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue32604'
versions = ['Python 3.10']

Linked PRs

• gh-76785: Use Pending Calls When Releasing Cross-Interpreter Data #109556
• [3.12] gh-76785: Use Pending Calls When Releasing Cross-Interpreter Data (gh-109556) #109586
• gh-76785: Module-level Fixes for test.support.interpreters #110236
• gh-76785: Add SendChannel.send_buffer() #110246
• gh-76785: Print the Traceback from Interpreter.run() #110248
• gh-76785: Support Running Some Functions in Subinterpreters #110251
• Revert "gh-76785: Print the Traceback from Interpreter.run() (gh-110248)" #110314
• gh-76785: Print the Traceback from Interpreter.run() #110322
• gh-76785: Clean Up the Channels Module #110568
• gh-76785: Add *Channel.is_closed #110606
• gh-76785: Make interpreters.*Channel Objects Shareable #110607
• gh-76785: Move the Cross-Interpreter Code to Its Own File #111502
• gh-76785: Crossinterp utils additions #111530
• gh-76785: Add an Internal ExceptionSnapshot Type #111572
• gh-76785: Return an "excinfo" Object From Interpreter.run() #111573
• gh-76785: Add Interpreter.bind() #111575
• gh-76785: Minor Fixes in crossinterp.c #111671
• gh-76785: Move _Py_excinfo Functions Out of the Internal C-API #111715
• [3.12] gh-76785: Use Pending Calls When Releasing Cross-Interpreter Data (gh-109556) #112288
• gh-76785: Clean up the Failure-Related _xxsubinterpreters Tests #112322
• gh-76785: Add _PyType_GetModuleName() to the Internal C-API #112323
• gh-76785: Fixes for test.support.interpreters #112982
• gh-76785: More Fixes for test.support.interpreters #113012
• gh-76785: Fix test_threading on Non-Subinterpreter Builds #113014
• gh-76785: Add Interpreter.prepare_main() #113021
• gh-76785: Show the Traceback for Uncaught Subinterpreter Exceptions #113034
• gh-76785: Avoid Pickled TracebackException for Propagated Subinterpreter Exceptions #113036
• gh-76785: Fix CODEOWNERS #113038
• [3.12] gh-76785: Make test.test_interpreters executable (GH-112982) #113470
• [3.11] gh-76785: Make test.test_interpreters executable (GH-112982) (GH-113470) #113473
• gh-76785: Improved Subinterpreters Compatibility with 3.12 #115424
• gh-76785: Update test.support.interpreters to Align With PEP 734 #115566
• gh-76785: Fix interpreters.Queue.get_nowait() #116166
• gh-76785: Simplify Channels XID Types #116318
• gh-76785: Minor Improvements to "interpreters" Module #116328
• gh-76785: Use PRId64 to Fix a Compiler Warning on Windows #116369
• gh-76785: Add concurrent.futures.InterpreterPoolExecutor #116430
• gh-76785: Expand How Interpreter Queues Handle Interpreter Finalization #116431
• gh-76785: Clean Up Interpreter ID Conversions #117048
• gh-76785: Drop PyInterpreterID_Type #117101
• gh-76785: Add PyInterpreterConfig Helpers #117170
• gh-76785: Consolidate Some Interpreter-related Testing Helpers #117485
• gh-76785: Raise InterpreterError, Not RuntimeError #117489
• gh-76785: Handle Legacy Interpreters Properly #117490
• gh-76785: Fix a Refleak in _interpreters.new_config() #117491
• gh-76785: Add More Tests to test_interpreters.test_api #117662
• gh-76785: Rename _xxsubinterpreters to _interpreters #117791
• gh-76785: Fix Windows Refleak in test_interpreters #117913
• gh-76785: Expand How Interpreter Channels Handle Interpreter Finalization #121805
• [3.13] gh-76785: Expand How Interpreter Queues Handle Interpreter Finalization (gh-116431) #121807
• [3.13] gh-76785: Expand How Interpreter Channels Handle Interpreter Finalization (gh-121805) #121811
• gh-76785: Minor Cleanup of "Cross-interpreter" Code #126457
• gh-76785: Minor Cleanup of Exception-related Cross-interpreter State #126602
• gh-76785: Improved Subinterpreters Compatibility with 3.12 (1/2) #126704
• gh-76785: Improved Subinterpreters Compatibility with 3.12 (2/2) #126707

[ericsnowcurrently]

In the interest of getting something landed for 3.7, so we can start using it in tests, I'm putting up a patch for a low-level interpreters module. In some ways this is a precursor for issue bpo-30439, which will add a proper public stdlib module in 3.8.

The module I'm adding draws from the ideas in PEP-554 (particularly for channels). Consequently, this will also give us an opportunity to try out some of the semantics from the PEP to give us better ideas for 3.8.

I expect to have some follow-on patches to facilitate simpler use in tests. This patch is big enough already. :)

[ericsnowcurrently]

@ned, it may be a little tight to land this given the time left before beta 1. However, this is meant as a tool for us to use in the test suite (particularly to test the subinterpreter C-API). So I'm arguing that, if necessary, it would still be okay to land this after the feature freeze. (I'm still hoping to get this in before the cutoff.) What do you think?

[ericsnowcurrently]

@nick, I may make the name change you suggested in issue bpo-30439 ("_subinterpreters").

[ericsnowcurrently]

FYI, there are a few things I need to clean up in the PR. However, I expect that those changes will be minor relative to the the whole patch, so I wanted to get the ball rolling on a review. :)

[ned-deily]

@eric, given the breadth of change introduced in the PR (including adding a new extension), I think it would be best if at all possible to get it in for beta 1 if we can resolve the review comments in time. If necessary and if there are no objections from other core developers, I would be willing to consider making an exception and allowing it into beta 2 as long as it remains a private interface. If it looks like it won't be in releasable shape by then, I think you should hold off for 3.8; doing otherwise would be unfair to others and to our downstream beta users / testers, for example, even if it is private, adding a new extension and setup.py changes potentially affect downstream packagers.

[ericsnowcurrently]

Sounds good, Ned. Thanks for taking a look. I should have everything finished up by Friday, so I'm hopeful for landing the change before the deadline. I may have a few minor tweaks to make after that, but I'll discuss that with you before making any changes if that happens.

[ericsnowcurrently]

New changeset 7f8bfc9 by Eric Snow in branch 'master':
bpo-32604: Expose the subinterpreters C-API in a "private" stdlib module. (gh-1748)
7f8bfc9

[ericsnowcurrently]

I've merged the patch without Windows support, which shouldn't be a problem given the purpose of the extension module. I've also added a PR for get the module building under Windows. I'd like to get that resolved ASAP.

[ned-deily]

Eric, looks like some buildbots are unhappy, for instance:

http://buildbot.python.org/all/#builders/13/builds/648

[ericsnowcurrently]

Yeah, I'm looking into it. Also, I noticed some refleaks that I'll be sorting out.

[ericsnowcurrently]

On 4 of the buildbots:

======================================================================
ERROR: test_drop_multiple_times (test.test__xxsubinterpreters.ChannelTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 890, in test_drop_multiple_times
    interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3)

======================================================================
ERROR: test_drop_single_user (test.test__xxsubinterpreters.ChannelTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 848, in test_drop_single_user
    interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3)

======================================================================
ERROR: test_drop_used_multiple_times_by_single_user (test.test__xxsubinterpreters.ChannelTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 957, in test_drop_used_multiple_times_by_single_user
    interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3)

======================================================================
ERROR: test_drop_with_unused_items (test.test__xxsubinterpreters.ChannelTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 899, in test_drop_with_unused_items
    interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3)