test_pickle/test_pickletools fail when running tests sequentially.

[Yhg1s]

When running the test suite sequentially (as is done as part of the release process), an interaction between test_imp, test_importlib and subinterpreters is causing test_pickle and test_pickletools to fail. The easiest reproducer::

% ./python -m test test_imp test_importlib test_pickletools
0:00:00 load avg: 0.22 Run tests sequentially
0:00:00 load avg: 0.22 [1/3] test_imp
0:00:00 load avg: 0.22 [2/3] test_importlib
0:00:03 load avg: 0.22 [3/3] test_pickletools
test test_pickletools failed -- Traceback (most recent call last):
  File "Lib/test/pickletester.py", line 1989, in test_builtin_types
    s = self.dumps(t, proto)
        ^^^^^^^^^^^^^^^^^^^^
  File "Lib/test/test_pickletools.py", line 11, in dumps
    return pickletools.optimize(pickle.dumps(arg, proto, **kwargs))
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_pickle.PicklingError: Can't pickle <class 'importlib._bootstrap.BuiltinImporter'>: it's not the same object as importlib._bootstrap.BuiltinImporter

test_pickletools failed (1 error)

== Tests result: FAILURE ==

2 tests OK.

1 test failed:
    test_pickletools

Total duration: 4.2 sec
Tests result: FAILURE

The failure happens in both test_pickle and test_pickletools (test_pickletools's traceback is more obvious), after in the same process running both test_imp and either test_import or test_importlib. Disabling test_imp.ImportTests.test_create_builtin_subinterp() makes the tests pass. (This was happening before #102982 as well as after it, so I don't think it is involved here.)

I expect this is an unintentional side-effect of the test, but this was not happening in the 3.12.0a6 release, and as far as I can tell the relevant tests haven't changed since before a6. I'm delaying the 3.12.0a7 release until it's clear this isn't a fundamental problem with subinterpreters.

Linked PRs

• GH-103247: bypass the import cache on the _require_loader helper #104186
• gh-103247: add tearDown to test_loader and clear module cache #104226
• [3.11] gh-103247: clear the module cache in a test in test_importlib/extensions/test_loader.py (GH-104226) #104345

[Yhg1s]

The problem is actually reproducible all the way back to 3.12.0a4, even though it was never caught in the release testing (I don't know why, yet). The problem was exposed by #100142, but that merely adds testing of picklability of builtins, which all things considered is not an unreasonable test.

[Yhg1s]

As it appears this is merely uncovering a long standing issue with a hole in subinterpreter isolation and importlib (the same test combination fails in 3.10 and 3.11), I'm unblocking a7 for now, but leaving this as a release blocker for beta 1.

[erlend-aasland]

I did some bisecting in 3.10 and 3.11, and it landed like this:

• In 3.10, first bad commit: 37dbbb2 ([3.10] gh-99578: Fix refleak in _imp.create_builtin() (GH-99642) #99644)
• In 3.11, first bad commit: 9dda902 ([3.11] gh-99578: Fix refleak in _imp.create_builtin() (GH-99642) #99643)

Strangely, I do not get the same result in main; the "parent" commit (cb2ef8b, gh-99642) that those backports were created from, reproduces the failure, as expected, but cb2ef8b2acbb231c207207d3375b2f8b0077a6ee~ (1cae31d) is not a "good" commit; the failure also occurs there.
The first bad commit I get in main is:

• 1f369ad (gh-98354: Add unicode check for 'name' attribute in _imp_create_builtin #98412)

The linked issues for those PRs contain some interesting history and debugging:

• gc_decref: Assertion "gc_get_refs(g) > 0" failed #99578
• create_builtin() in _imp module trigger segfault if taking a builtin object as input #98354

(It would be nice if someone else could bisect on their machine and verify my findings.)

[sunmy2019]

even though it was never caught in the release testing (I don't know why, yet).

Is it because the release testing in running in 4 processes?

[Yhg1s]

No, the release testing is done sequentially, and that hasn't changed in 2 years: https://github.com/python/release-tools/blame/master/run_release.py#L486